John's research

Intro

Calling this research is probably a bit wild. Anybody with better writing or organizational skills, feel free to tweak. If there are stuff missing, either add or tell me.

Here I'll try to capture the work done with our FreeBSD based wireless routers. Currently they are a mix of Soekris net4501, net4526, net4801, Wrap 2C, 1E, Gateworks Avila and ADI Metro Pronghorn boards.

How the config files on our distro fit together

FreeBSD is a lovely operating system and the standard startup scripts already cater diskless systems, read-only filesystems, network boot with a nfs root, etc. The idea was to try and leverage as much of FreeBSD's startup scripts and procedures as possible.

This makes for less work and more testing, at least for the common parts. The standard FreeBSD startup scripts already cater for a diskless case, where /etc and /var are ramdisks and the rest can be read-only. That fits well with our case where we want the Compact Flash (CF) mostly read-only for reliability, but still needs things in /etc and /var to change during and after bootup. So we force this by creating the file /etc/diskless.

So maybe a description of the boot process is in order. I'll skip some detail that I think is not important to understand this. The kernel will at some stage get bored configuring RAM and probe for devices and look for something else to do. It will then load and run /sbin/init. Init will do some things and then run the /etc/rc script. And this is where the interesting stuff begins.

rc will try to determine if it is the diskless case. If so it will run /etc/rc.initdiskless, which will create a ramdisk over /etc and populate it from /conf/base/etc and then from /conf/default/etc. The idea being that you can have a standard /etc tree in /conf/base/ and your local changes in /conf/default/. So all the files that we change will be in /conf/default/. After that, rc.initdiskless will also create and populate a /var ramdisk. When finished, it will return to rc.

FreeBSD startup scripts reside in /etc/rc.d/ and /usr/local/etc/rc.d/ for packages. Now rc will use rcorder on those two directories to determine the dependencies and order they must be run in. /etc/rc.d/early.sh is one of the first to run. It will run /etc/rc.early if it exists. It does not exist by default.

Most startup scripts in rc.d/ can be enabled/disabled in /etc/rc.conf. Maybe that is a bit too simple. They will use a function called load_rc_config to load /etc/defaults/rc.conf, /etc/rc.conf and /etc/rc.conf.local, in that order. These files just contain shell variables for example:

hostname="arm-tst.mesh"
mesh_dns_enable="YES"

So for our distro we install /etc/rc.early where the biggest chunk of our configuration will happen. A lot of it is just to put the right stuff in /etc/rc.conf and other app specific config files. Then the rest of the normal FreeBSD startup procedure, to run the startup scripts in rc.d/ will take care of the rest.

So the most important config files are:

• rc.conf - edited by auto script if auto_update_enable and _url set
• rc.conf.mesh - included early by rc.conf with defaults for the mesh
• rc.conf.local - overrides with partly manual conf

The config directories:

• /etc - ramdisk with end results, everything is lost on a reboot
• /conf/base/etc - original FreeBSD stuff
• /conf/default/etc - changed files, our and packages
• /usr/local/etc - a symlink to /etc/usr.local.etc

The basic idea is that config overrides that are done manually, have to be done in /conf/default/etc/rc.conf.local because it is read last and so overrides previous settings.

The startup scripts will create the ramdisk /etc, then recursively copy /conf/base/etc into it and then /conf/default/etc. The real /etc needs diskless, rc, fstab and rc.initdiskless in it for this to work.

Files that will need to be twiddled by scripts on startup:

• /etc/rc.conf - interface conf
• /usr/local/etc/olsrd.conf - interface, dns
• /usr/local/etc/dhcpd.conf - dhcpd
• /etc/resolv.conf - ?
• /usr/local/etc/dnsmasq.conf - dhcp + dns ?

There are probably more.

What is there to tweak

In theory we want to end up tweaking / configuring nothing, but we are not there yet. :-)

These configs go in /conf/default/etc/rc.conf.local unless otherwise specified. You can look in rc.conf.mesh to see what the defaults are, but do not change it there. That will break upgrades. Remember that the CF is normally read-only, so has to be made read-write before editing the file with

mount -u -w /

And made read-only again afterwards with

mount -u -r /

A network needs one box that is the dns server. On that box you will need to set

mesh_dns_enable="YES"

One thing that is often needed is to swap channels on the BackBone links. You can change it with lines like this:

mesh_bbchana="140"
mesh_bbchanb="136"

If you want to give the box a name:

hostname="ajay-rtr.cids.org.za"

If you need to override everything for an interface:

ifconfig_ath0="mode 11a mediaopt adhoc channel 165 ssid ptabb"

A specific prefix for an interface:

ipv6_prefix_ath0="fd9c:6829:597c:10"

If you need to override the ipv4 address:

ipv4_addrs_npe0="10.3.2.1/24"

Common files that also might need to be tweaked are:

/conf/default/usr.local.etc/olsrd.conf
/conf/default/usr.local.etc/dnsmasq.conf

Although, if you need to, tell me so that we can see if it can be handled automagically by a script.

How bits and pieces work

Upgrades and Automagic upgrades

Disk split in 2 partitions. Upgrade the one that is not active.

Auto script run from cron to check for updates.

/upgrade tool used for manual updates and by the auto script to do the actual updating.

4over6 Automatic tunnels

An idea to tunnel IPv4 packets over our IPv6 mesh networks without having to configure each tunnel. Think of it as a kind of one to many tunnel. So only one need to be configured on a host.

Assumptions

The IPv6 part will use OLSR or some routing protocol where static routes can be injected. The host doing the tunneling needs to be dual stacked. The edges will have IPv4 subnets from the 10.0.0.0/8 network in addition to the IPv6 subnets.

The way it works

An IPv6 subnet address (/64) is chosen that will be used as the tunnel prefix throughout the network. Each tunnel box must be configured in the following way:

• Its "local" interface(s) is/are configured with a 10.0.0.0 subnet(s).
• A virtual tunnel interface (tunX?) is created.
• An IPv4 static route for the whole 10.0.0.0/8 subnet is added into this interface. (XXX Maybe default route?)
• An IPv6 static route is calculated from the chosen IPv6 tunnel prefix plus the local IPv4 subnet(s) (fdfd:efef:abab:cdcd::10.1.2.0/120)
• This static route is also given to the routing protocol to announce.
• The 4over6 tunnel daemon [*] is attached to the tunnel interface. It is configured with the IPv6 tunnel prefix.

The 4over6 tunnel daemon

The configuration it needs is the IPv6 tunnel prefix. It receives IPv4 packets (from the local interface(s)) convert the header to IPv6 using the IPv6 tunnel prefix and sends it out again. It receives IPv6 packets, convert the IPv6 addresses back to IPv4 and sends it out again.

   |------------------|                         |------------------|
   |       host1      |                         |       host2      |
   |                  |                         |                  |
   |       |---|      |                         |       |---|      |
   |       | B |      |                         |       | E |      |
   |------------------|     /\/\/\/\/\/\/\      |------------------|
       A |       | C       /     IPv6     \         D |       | F
---------|       |--------< Routed Network >----------|       |----------
                           \              /
                            \/\/\/\/\/\/\/

Host1 and host2 are connected via an IPv6 routed network. (No native IPv4) prefix used for 4over6 fd9c:6829:597c:fefe/64

host1
A - 10.1.1.1/24
B - virtual (tun) interface. Static routes 10.0.0.0/8 and
    fd9c:6829:597c:fefe::10.1.1.0/120 pointing into interface.
    A route fd9c:6829:597c:fefe::10.2.2.0/120 pointing in host2's direction.
C - IPv6-only address. 

host2
D - IPv6-only address.
E - virtual (tun) interface. Static routes 10.0.0.0/8 and
    fd9c:6829:597c:fefe::10.2.2.0/120 pointing into interface.
    A route fd9c:6829:597c:fefe::10.1.1.0/120 pointing in host1's direction.
F - 10.2.2.1/24

Installing FreeBSD on a Avila/Pronghorn Metro ARM board

Gateworks Avila boards

Note! FreeBSD detects the mini-pci cards different from the labeled order on the PC board. It detects it in this order: J3, J1, J4, J2 So the card in J3 will be ath0...

ADI Pronghorn Metro boards

Note! FreeBSD detects the mini-pci cards in reverse order from the labeling on the PC board. So it will detect from PCI3, PCI2, PCI1 to PCI0. So the card in slot PCI3 will be ath0.

• Redboot use 115200 baud, so use

> cu -l /dev/your-serial-dev -s 115200

• Break RedBoot with ^C where it says you can. :-)

> ip -l 10.0.0.234/24 -h 10.0.0.1 # LAB C155
> ip -l 146.64.5.234/24 -h 146.64.5.1 # JHAY-LAB
> load boot2

• Now write it to flash.

> fis create boot2

• On the ADI Pronghorn boards you might get an write error. They seem to lock the flash. Just unlock it. Look for the address in the error message and do something like this and do the fis create command again:

 > fis free
 > fis unlock -f 0x50060000 -l 0x20000

• Now go and configure RedBoot to automatically run boot2 on startup.

> fconfig

• Change the "run script at boot" to true
• Change the startup script to:

fis load boot2
go

• Change the timeout to 3.
• Change BOOTP enabled to false.

• Enter a . on the next option to end fconfig. Choose y to save it.

> load -b 0x200000 kernel-avila
> go

• Type <enter> at the shell prompt. It should list all the distros available. Choose the correct one for your architecture.

> ./writedisk disk-arm-7-20070315.tgz # Normally use the latest
> reboot

• Reboot to make sure everything is working.

Installing a distro

Upgrade

Well if it is only an upgrade, ie. the box is already running an older version, just make sure it is connected to the network and type (as root):

/upgrade ftp://crypton/disk-arm-7-20080813.tgz

Substitute the file name for the correct one. The upgrade script understand ftp, http, ssh and a local file:

/upgrade <user@host:file>
/upgrade <ftp://[user[:password]@]host[:port]/path/file>"
/upgrade <http://[user[:password]@]host[:port]/path/file>"
/upgrade </path/file>"

New Install

This is a bit more involved.

• For the ARM boards, you will need a tftp server loaded with boot2 and an ARM kernel with nfs booting compiled in. You will also need a nfs server setup with an extracted FreeBSD ARM tree.
• For the i386 based boards, you will need a dhcp server that can handle PXE (for the soekris), with a tftp server and a nfs server.

The basic idea is then to nfs boot the board into single user mode and then run the writedisk script which will prepare the disk and install the distro on it.

Building a distro

This one is easy. If you have root on my box, you go to /home/jhay/small and type:

./build_chroot small-7-arm.cfg

to get an ARM distro or

./build_chroot small-7.cfg

for an i386 distro. A tiny distro that can fit on half of a 64M CF can be built with:

./build_chroot small-7-32m.cfg

The trick is to get root on my box or to get me to document more. :-)

Todo

Ideas that might get implemented.

• Handle extra hna{4|6} entries from rc.conf
• Should updates be signed so that the auto update will only update to a signed distro file?
• A wacky idea for updating even in the face of incompatible routing protocols. Have every box run a tcp relay to an update box. So every box can connect to a neighbor's relay for updates. This will not solve the problem of incompatible links, eg. adding wep or changing ssids.
• Robust addressing scheme for big networks that need to cooperate.
• What else?