Community wireless networks
Groenkloof mesh
For the last year or so, I've been using some of my spare time to help with the creation of wireless community network in our suburb. This exercise helps me to get firsthand experience all the issues associated with community wireless networks and it lets me get a better understanding of normal network users and their needs.
The network started with a bunch of friends that all know each other, but when the word got out and a lot of other people wanted to join the network as well. Our network has now grown to almost 20 Nodes, running on Linksys WRT54GL routers with Freifunk firmware. We have a community backup server on the network where people can make an off-site backup of data. Internet connectivity is available via a 4Mbps ADSL link at one of the users homes. The guy with the ADSL link is also an active gamer and his only concern was that the community network will not affect his gaming.
Up to now it was unmonitored and free (sponsored by the guy with the ADSL link) to all the users. At this stage we have reached a point where we need to find a way for users to contribute to the costs of the Internet connection and to make sure that the rest of the network does not influence his gaming.
Live drawing of the nodes in our network: [1]
We've decided on the following:
Log and count all Internet traffic to get some Internet usage statistics Get a list of email addresses and telephone numbers for all the users Create a club and let all the users join it Open a bank account for the club (used for user deposits and Internet expenses) All contributions/payments are voluntary We will have two types of users: free-users and paying-users Paying users will have full Internet access at full speed Free users will not be capped, but their bandwidth to the Internet will be reduced to xKbps Free users will not be restricted inside the community network Build our own Internet Gateway that can split traffic between SA traffic and International traffic Open two ADSL accounts: One SA only and one full Internet account Find a way to provide some quality of service for the gaming ports
For now we will implement these measures and then see how it turns out. My responsibility is to work on our new Internet gateway and the following discuss some of the functions of our new gateway.
Splitting SA traffic and International traffic
Why do we want to split the traffic, well it makes good sense if you look at the difference in pricing between a local SA only ADSL account and a normal (SA+International) ADSL account. One can get a 1 GByte local SA only account for R10-R20 while normal 1 GByte account can be anything from R70-R120 depending if it's shaped or unshaped.
The first thing one need to do is to find a list of all the IP networks that are being used in South Africa. Then you need to buy two ADSL accounts, one SA-only account and one International account. Most of the ADSL routers don't have the ability to handle two different PPP accounts at the same time. This means that the ADSL router should be configured to bridge all PPP traffic, while our new router needs to handle two PPPoE accounts. Then you need to route the traffic according to it's destination. Here is a couple of methods to split the traffic.
On Windows there is a program called Route Sentry. This program will split and route the local and International traffic for you via two different PPPoE accounts. This method limits you to a specific PC only.
Make use of IPcop in Linux. Haven't looked at this option because I'm using FreeBSD.
Get hold of a list of SA only IP subnets and then set-up static routes according to list of SA subnets.
Beg some ISP to give you a BGP feed. Ok, we don't have an ASN number, but one can make use of a reserved-for-private-use ASN number. Unfortunately ISP's don't like to give private people BGP feeds.
I've decided to implement static routes according to a list of SA only subnets. SA subnets can be retrieved from public SA route servers. A Route Server serves as BGP information exchange to other BGP routers. Unlike normal BGP router, Route Server must have several routing tables for managing different routing policies for each BGP speaker. Some of these Route Servers allows public access via telnet. E.g.
Public routes servers in SA.
routeserver.ginx.org.za tpr-route-server.saix.net route-server.is.co.za
telnet route-server.is.co.za terminal length 0 show ip bgp exit
Here is a perl script that I wrote to get all the SA routes from route-server.is.co.za. Then it adds a mask to routes with default mask and it filters out routes with smaller masks if one with a bigger mask already exist.
Here is the new SA routes file that I've downloaded.
Here is a nice link that explains how to do it on Linux [2]
Building our GroenWifi router (v1)
The first step was to find a suitable hardware platform. I've looked at various options and in the end I decided to use a Soekris 4801. The 4801 is a embedded i386 compatible device with three Ethernet ports + I can run FreeBSD on it. See [3] for a full description of the hardware platform
Initial network setup
My initial idea was to make use two ADSL routers, one with a SA-only ADSL account and the other with a normal SA+International ADSL account. The soekris box will then slit the traffic and forward it to the correct ADSL router. This is one of the big reasons I've selected a hardware platform with three Ethernet interfaces (Mesh, SA-only and International)
picture ?
Installing FreeBSD on the 4801
Quick job, just took our standard mesh-router distribution and installed it on the 4801 (complements to John)
Configuring to box to split the traffic
Not so difficult. I've added all the SA local subnets into Quagga (routing daemon) and directed it to my SA local interface, while pointing my default gateway to the other interface (International). I did a quick test with two NAT boxes and it worked very well. With the aid of traceroute I could see that International and SA traffic would take different routes to the internet.
picture ?
Add Accounting for Local and International traffic
Now it becomes more interesting. The easiest method I know to get some stats is to use the counters of some IPFW rules. Then one can do a "ipfw show" to read the counters. The fist step is to split the rules based on interface name and the direction of the traffic. e.g.
00200 skipto 1000 ip from any to any in recv sis0 00300 skipto 2000 ip from any to any out xmit sis0 00400 skipto 3000 ip from any to any out xmit tun1 00500 skipto 4000 ip from any to any in recv tun1 00600 skipto 5000 ip from any to any out xmit tun0 00700 skipto 6000 ip from any to any in recv tun0
That way one can count the traffic going to, and coming from our International interface, as well as the SA interface, each subnet that we use in the mesh. This way I get four counters/household on the mesh (Internat-in, Internat-out, SA-in, SA-out). E.g. counters for SA-in and SA-out
03001 51 6069 allow ip from 10.4.3.0/24 to any 03002 3549 174403 allow ip from 10.4.4.0/24 to any 03003 0 0 allow ip from 10.4.5.0/24 to any 03004 347 41661 allow ip from 10.4.6.0/24 to any 03005 169 7680 allow ip from 10.4.8.0/24 to any 03006 0 0 allow ip from 10.4.9.0/24 to any 03007 0 0 allow ip from 10.4.10.0/24 to any 04001 0 0 allow ip from any to 10.4.3.0/24 04002 5181 4248108 allow ip from any to 10.4.4.0/24 04003 0 0 allow ip from any to 10.4.5.0/24 04004 471 504541 allow ip from any to 10.4.6.0/24 04005 178 53005 allow ip from any to 10.4.8.0/24 04006 0 0 allow ip from any to 10.4.9.0/24 04007 0 0 allow ip from any to 10.4.10.0/24
Now one just need a script to add the in and out traffic per subnet/household
Add Bandwidth management
Another great feature of FreeBSD is it's build in support for Dummynet. Dummynet is a great traffic shaper, bandwidth manager. We decided that we have basically three types of users in the network.
Free user - They get free access to the network, but their bandwidth is limited. Paying user - They pay a monthly fee and they share the available bandwidth (using fair queuing). Gamers - A paying user, but with some reserved bandwidth for gaming
Dummynet configuration is integrated into ipfw. You first have to define a pipe. Then you configure that parameters for the pipe e.g. 50kbit/s etc. Then you attach a pipe to a subnet. Remember, you should have one pipe for outgoing and another for incoming traffic. E.g. ipfw rules
#Bandwidth allocated for Wireless Mesh pipe 1 config bw 448Kbit/s pipe 2 config bw 3584Kbit/s pipe 3 config bw 25Kbit/s pipe 4 config bw 25Kbit/s
#paying people get a queue weigth of 100 queue 1 config pipe 1 weight 100 queue 2 config pipe 2 weight 100
#free people get a queue weight of 20 queue 3 config pipe 1 weight 20 queue 4 config pipe 2 weight 20
#adding queues to subnets 01001 queue 1 ip from 10.4.3.0/24 to any 01002 queue 1 ip from 10.4.4.0/24 to any 01003 queue 1 ip from 10.4.5.0/24 to any 01004 queue 1 ip from 10.4.6.0/24 to any 01005 queue 1 ip from 10.4.8.0/24 to any 01006 queue 1 ip from 10.4.9.0/24 to any 01007 queue 1 ip from 10.4.10.0/24 to any 02001 queue 2 ip from any to 10.4.3.0/24 02002 queue 2 ip from any to 10.4.4.0/24 02003 queue 2 ip from any to 10.4.5.0/24 02004 queue 2 ip from any to 10.4.6.0/24 02005 queue 2 ip from any to 10.4.8.0/24 02006 queue 2 ip from any to 10.4.9.0/24 02007 queue 2 ip from any to 10.4.10.0/24
For more info on Dummynet follow this link [4] and this one for FreeBSD firewalls [5].
Testing everything
I've used the same setup as the one that I use to test the splitting of network traffic. See picture. With the ipfw show command I could see how the counters would increase when traffic was send over the different networks. To test the bandwidth shaping I've did a file transfer from different subnets behind the router to a ftp server on our network. Works great, OK, after fixing some bugs :-)
picture ?
Next step was to test it with the actual ADSL routers that we planned to use in the network. Got host with one of the routers and started to play with it at home. I don't have an ADSL line at home, but I can test the Ethernet side. Now for the fist hick-up. The router had limited features and it did not support routing or static routes. This meant that the ADSL box will not know how to send traffic back to the mesh. OK, back to the drawing board to look at all the options (Another ADSL router, double NAT, pppoe on FreeBSD etc).