Mpumulanga Mesh

Location

White River is a medium sized town about 20 minutes north of Nelspruit, the capital of Mpumalanga Province. Peables Valley and the Masoi Tribal land is located 20 minutes to the North East of White River along the road leading to the Kruger National Parks Numbi gate.

The area is hilly with some large granite outcrops. The Peables Valley stretches from the ACTS clinic and divides the commercial farms from the Masoi area. Commercial farming in the broader area is predominately timber but fortunately along the valley there is no timber farming (this would pose a connectivity problem and impact line of sight requirements). The farmers as a community are already well connected (organisationally not in terms of Internet connectivity) with commando like structures in place because of crime.

The ACTS clinic is located at the beginning of the Valley. ACTS stands for AIDS Care Training and Support Clinic. They provide medical services to AIDS patients, councilling, testing and Anti-retroviral (ARV) treatment. The clinic has gained a reputation within the community with many people coming from outside of the Masoyi area for treatment. They are have been selected for managing ARV roleout because of their well developed councilling, testing and support group networks. Currently they have about 200 people on ARVs with this figure likely to grow to over 600. These treatements are being monitored by researchers at WITS University.

The Masoyi area runs along the right hand side of the valley. We identified approximately 10 schools, with two identified as being high schools with computer facilities. The project team has good links to a teacher in one of these schools who is in charge of the computer lab. In the area there is also a teachers training centre, government hospital and various government clinics in the area. The ACTS clinic runs a hospice close to one of the Government clinics.

Situation in the Community

The community is underserviced. Most roads remain unpaved with the main road having been rebuilt after the floods of 2000. Electricity is available within the community. The community is poor and has been hugely impacted by AIDS. Losing salaried member of households through AIDS has a big impact on extended families. There is also the associated superstition and secretiveness around AIDS which does not help improve the situation.

ACTS clinic

The clinic is a non-profit organisation established to provide AIDS care in the Masoyi area. It was initially sponsored by Glaxo Smithcline. The clinic has two doctors, an onsite hospice, training centre and accomodation for people on training courses.

A number of staff have been trained to provide pre-test and post-test councilling. This leaves the doctors free to focus on the medical aspects of treatment. The clinic operates in many ways like a private doctors practice. They charge consultation fees and patients on medical aid are charged accordingly. They however usually waive all fees as soon as it is established that a patient is suffering from AIDS. The clinic is relatively sopphisticated in their use of computers: they run practice management software, are connected to the Internet via Vsat for their research with WITS. They also have some custom MS Access based applications written internally to track patient statistics.

Proposed Solution

Deploy a mesh network within the Peables Valley, run training courses to transfer skills on mesh setup and installation, work together with the clinic staff using Free Attitude Interviews (FAI) to discover applications and uses that work for the staff.

What problem is being addressed

The main thrust is to connect the clinic to its community hospice and thus allow them to extend their computer facilities to the hospice (practise management software and statistics gathering). This is most easily achieved through a wireless network and more specifically a mesh because of the terrain.

A mesh network requires a number of nodes in areas not owned by the clinic but that are part of the commuity: schools and farmers. The main thrust of the research question is to implement the network in such a way that the firstly the clinic and secondly the community have taken ownership of the network. That they have created systems that ensure that the infrastructure is maintained and expanded by the users of the network.

Technology Explained

A wireless network involves radios and aerials. The radios broadcast and receive signals which are boosted and directed by the arials. Anyone with a radio can access the network. The spectrum used for Wifi required line-of-sight for connectivity. The routes between sites are also hard coded or manually adjusted in each device. This makes it technically difficult on two levels. Firstly, you need a deaper understanding or the radio spectrum and the operation of radio wave and secondly, configuration of the equipment requires a higher level of skill.

A mesh network uses the same infrastructure: aerials and radios. But at the network level it operated differently. The network simply requires that each node can see at least one other node (ie it can be part of the network) and that each node can send traffic to any node on the network (ie there are no islands which are not connected to the whole). In a mesh network this problem is solved at the software level. As long as your node can see another node it will be able to determine the routes to follow. The also allows for a level of self healing and thus less maintenance. If for example a truck parks in from of the node or a tree grows up over time and block the signal to another node then as long as the node can see an alternate node it can route traffic via this new path. On a conventional wireless network this would need to be manually configured.

The choice of mesh in the Peables Valley was made for these reasons: Hilly terrain that requires a number of hops to reach the destination Low skilled network operators Changing conditions in the environment. Nodes that disappear through neglect, theft, obstruction.

Social Challenges

There are a number of social challenges:

Firstly, for the the network to operate to serivice firstly the clinic and secondly the community it requires the cooperations of teachers in schools and of farmers. They all have there own agenda's there own perceptions of their expected level of involvment and their expectation of how aid is given to them.

In the school we are concerned about a hand-out mentality that will not see the school working hard to be involved in the extension of the mesh but simply being involved as long as it takes to link the school to the Internet. There is also the risk of our contact at the school wanting to be seen as being in charge and not wanting to work together with schools in the community.

To addess that we are looking at bringing the school pupils into the process from the beginning and teaching them how to install and configure equipment.

Secondly, with the farmers we are confident that it will be easy to sell them the concept of Internet connectivity. However, for the network to work for them will require that they connect to schools in the Masoyi area. We are not certain about the realtionship between these two groups. Considering the crime levels there is the risk that there is not a good level of trust.

Our strategy is to involve people from both groups during training workshops. Which at the very least would allow realtionships to be established.

Lastly, this is not the primary focus of the ACTS clinic so there will always be the risk that people are redeployed.

However, our main contact in the clinic is an electrical engineer by profession and the clinic itself has proved that they are a teaching organisation (their previous garder now runs the reception).

Technical Challenges

Mesh networks are a new phenomena with the software sometimes not yet at the level required for the mesh to operate well. There is the risk that in this project we will discover the limitations of the technology and have to spend time compensating for that.

Some of the endpoint nodes we have identified require many hops. This requires many nodes. This is both a social and technical challenge. Social in that the project needs to bring each of those players on board and technical in that each of those nodes needs to be built, deployed and maintained.

There is also the risk of theft. The node equipment has very little commercial resale value in the community, although there seems to be wifi operating in the Nelspruit, White River area. However, that never seems to be a consideration of petty thieves. Hopefully deploying the equipment in established locations: schools, shops, private homes and not simply in the open will help secure the equipment.

Equipment Ownership

As the objective is to create a community based network where the network is seen as a community asset so therefore some of the equipment at key nodes should belong to the community. Official ownership will be with the ACTS clinic until such time as a more formal structure for ownership of key mesh equipment can be established.

As the mesh grows individual node owners will own their own equipment. So a farmer or school will be the owner of that equipment and it will form part of the mesh. It is hoped that farmers will partner with institutions such as schools in which case the partners will define ownership relationships.

The project does not envision supplying any equipment to farmers unless they are located such that having equipment on their farm is critical to creating the mesh. In such as case we would look at funding that node if the project has to but the project will still attempt to have the farmer install his/her own equipment.

Intellectual Property

The project will release all work under a suitable Open Source license. This work includes documentation and results of the research.

The project will not be developing any independent software but will be using other freely available mesh software. In the case that software is improved or modified it will be released under the license of the original software project to ensure that it can be included in future releases of that software.

Sustainability

A key component to sustainability will be involvement of the farmers, Bible colleges, etc within the Peables Valley. The project will draw in the farmers for the following reasons: 1) they are a source of skills, 2) they have finances to pay for Internet connectivity and 3) that once they become users they are likely to become dependent on the mesh and thus ensure its ongoing success.

Part of the program will be to educate participants from the start so that they can extend the reach of the mesh. Included will be education of pupils from the partner schools. This education is designed to ensure that enough skills are available in the valley to repair nodes, setup new nodes, manage the mesh and train others.

A key component to the sustainability is that the project manager as the ACTS clinic is a qualified electrical engineer. We are therefore confident that it will not be difficult to educate him on the workings of the mesh network. He has already demonstrated his excitement and commitment to the project by spending a Saturday with the project leaders evaluating high sites. He is involved in training staff and is extremely practical so will be invaluable in erecting high sites.

Pros and Cons of the system

First considering some positive aspects of the mesh. Firstly the clinic is already using IT infrastructure in the following areas: ARV monitoring, clinic management and accounts. Farmer, bible colleges, etc in the community already use the Internet for email etc but usually at great cost because of the limited choice of commectivity options. The partner schools already has a lab that has in the past been connected to the Internet. So there is already a body of people who have demonstrated the willingness or need to use the Internet who we feel would be willing the use a mesh if it provide always on and lower Internet connectivity costs.

The potential problems we see are the following. There is the risk that the effort to manage or be part of the mesh may be more expensive then direct connection especially in the case of the lack of good neighbourliness from any mesh participant. The clinic, although they currently use IT extensively, may in fact not need to extend that usage beyond the main clinic. The risk with the schools we feel is that there is a general hand-out mentality and that there will not be the will to learn, extend the network or integrate this into the curriculum. Another large risk is the sustainability of the network not from the mesh side but from the side of financing the hop onto the Internet itself.

Backhaul Connectivity

The connectivity to ACTS is provided via Telkom VSAT. ACTS uses the connection as part of their involvement in ARV roleout for the Right To Care group. They have an Internet based application for capturing details of patients for monitoring and evaluating patients receiving ARVs

Network Topology

September 22, 2005, WiFi in the bush or the plot

Duane visited the IDRC funded workshop for WiFi in Africa. Sorry don't know what the real project name is. Sat in on the session on regulation, the Department of Communications delegates decided that this would be a good time to leave. Nice when you know you can't actually stand the heat.

Had a look at their cantenna based access-point liming over about 5km. With a Linksys strapped to a tree with... red tape... man we can't get away from red tape in this industry.

Most surprising was plot.net a local WISP that delivers Internet to about 100 customers and nobody new that they existed before booking the venue. They do long haul links to ADSL connections in Pretoria. Quite an innovative solution to high cost access. I want to explore that possibility in Peebles Valley.

October 06, 2005 Wikipedia on the mesh

David and I visited the Peebles Mesh this week for two days and here is a quick synopsis of events:

- We installed a firewall. This has software which should allow us to control and cap bandwidth and it separates us from the clinic. - We installed a local copy of Wikipedia. Still some bugs but very fast - We can now access the site remotely

Big SURPRISE

Our cantennae was still operating! We did a data transfer from the wireless box to ourselves. 3 hops I think at 192kB/s

First farmer online

We connected the chairman of the local farmers group to the Internet. It was lovely we just watched and took notes. The farmers younger brother had returned from the UK with a new laptop. We got that connected. The first thing he asked is if he could download music - ho hum :)

The sister of the farmer was also visiting and started giving her mother a sales pitch about how they could talk to each other using skype for free.

Was lovely to watch and we had boere kos but arrived a bit late for lunch. We will ensure that doesn't happen again.

July 06, 2005, Realisation of a mesh vision

Today we had a film crew filming the installation of a mesh node. It was great fun but they did waste our time a bit. Technically this is what we got done:

   * Installed the node on Agnus' house permanently. Lassy did all this work
   * Put a 3m pole at the Hospice to see if we can get over the ridge
   * Upgraded AGNUS, HOSPICE and SIKILE to OpenWrt
   * The clinic is cabled to various buildings so we will be able to recover one node and just run a cable from the FLATS node to the VSAT.

We had a 7ms delay between AGNUS and ACTS_VSAT which is a route that passes through all present nodes. VoIP is now truely achievable. Wonderful performance compared to WDS mode. Throw out WDS wherever you see it its just not worth the effort. It is honestly more simple to setup OLSR. Glad we've learnt that and can move on.

We still have the following technical hurdles to overcome. The default route keeps disappearing. We need to install a proper firewall. Once that is done we can move onto the community adoption.

The day was full of some lovely experiences of the potential of what we are doing.

Once we had AGNUS up we setup Dave's laptop and had Agnus' daughter and a neighbour sit at the computer. Dave opened Google. They didn't know what to do. We asked them what information they wanted to know. Stuck again. "Who is your favourite singer?" - Alicia Keys - "OK type that in the search box, then click on search" - Shouts of glee as the search results return. "Now click here". Off they went to Alecia Keys home site and disappeared for about 10 minutes. Does that count as training? The two of them were sitting inside quite a dark house, they sat together on a sofa and the laptop screen glowed in the semi-dark with their face highlighted. This exact image was the vision I had for this project. So that made my day.

I did a quick training course in the dirt. Explaining what the different aerials did, how it all fitted together. And of course how to use a high-gain antenna as a fish braai. Lassy provided Swati translation as soon as there was any confusion.

Now off to the school to film students working on computers. They wanted to interview the teacher, who was a standing in for Mr Sihlabela who we usualy work with. "What will you use the Internet for at your school", to which he replied, "I don't know". David did a quick exercise of taking the teacher, a history teacher, to Google. They Googled for "Shaka Zulu" and ended up at a very good site on the subject. "What will you use the Internet for at your school", and we got a 5 minute reply. Does that count as training?

What worked best about this trip was having a very focused objective - get the mesh network working. We didn't care about the firewall, the billing, shapping, etc. And we got it done. The next trips will be similarly technically focussed.

July 05,2005 Moving on to OLSR

I left Peebles Valley at the end of May and this is the first return since then. David and I are migrating all the Linksys boxes from Sveasoft to OpenWrt, specifically the builds produced by Freifunk which has a nice web interface. Lassy has joined us again and is helping us move the equipment around.

The task involves loading the Freifunk firmware onto the box. This requires local connectivity. We still had connection problem until we discovered the ghosts of a WDS past. Alot of WDS settings were still on each of the boxes. We've now had to go and remove those especially the Lazy WDS mode settings which keep establishing bridges. OLSR looks like the right solution. Its the same amount of work as WDS setup but you get real routes, ones that you see and can traceroute through so you can actually see exactly how packets are being routed on the network. I'm much much happier. Yesterday we got USAID, ACTS_VSAT and ACTS_FLATS converted. Today we are in the field with a film crew and will migrate: AGNUS house, SIKILE_HIGH and FARM_COBUS. If we can get traffic from either Agnus' house or Cobus' farm then we know that the protocol is good, both these points would have to route through. SIKILE, USAID, ACTS_FLATS then ACTS_VSAT. WDS could never do that and if we can get this working then the mesh is good and we can move onto the next challenging problem.

At the last conservancy meeting at which ACTS clinic was present they mentioned the Mesh but all the farmers were wondering when exactly anything would happen. Hopefully we have a working system before we leave.

We also spoke to Frans Benz whose property is below USAID and he is still interested.

Of the nodes that we have observed all were powered up except ACTS_VSAT this was good as it means that people are leaving the equipment to do its job. THe VSAT was down because the plug is just too close to a useful double adapter.

We have brought 300m of CAT5 cable which the clinic will use to run cable from the VSat to the Flat so we can eliminate one node and use it elsewhere. This clinic has arranged conduit and pulling of the cables.

February 19, 2005, Lessons learned from week long meshing

Lessons learnt

   * Test test test: test devices in a known controlled environment. The box couldn't connect and we weren't sure exactly why.
   * Use your own stuff: don't use other peoples stuff. We tried to use the alarm companies mast, it was rubbish and about to fall down. But it could also endanger our equipment when the alarm company removes there antennae.
   * Equipment: cover for your laptop so you can see the screen. Backpack/fishing jacket also a head- mounted display wouldn't hurt (to monitor the wireless network). Rifle-mounted antennae with sight. Tri-pod for temporary mounting of equipment. Ladders are a problem you need a long extension ladder about 6m.
   * Windows Sucks: we had major packet drop yet on Linux the network pings work perfectly. Either Windows has a simple and incompetant network layer or it puts so much rubbish on the network that it sucks up all bandwidth. Why do people use this rubbish!
   * Testing protocol: It would be usefull to have a set standard of how you tackle problem. Many time you go down the wrong route. Simple idea: mount antennae with laptop and get signal strengths right. Then connect box browse to the box over the wire and check for links. Lastly connect to the box as a wireless client. 

Observations on the first stage

The object of this exercise was for David and I to learn how to setup the equipment before involving the community. This objective has been met: the mesh functions well, we can monitor it and we can use it from any node. Successes

We created a 4 node mesh that works and can connect to the Internet. Although we had issues these were just teething problem for the technicians and the hardware is really box-drop ready. This is really positive for a non-technical mesh community.

We met with the farmers and they now know what we are doing which give us easier access to their properties. The USAid couple were very helpful, very keen and are well known and liked in the community. We hope that this relationship and them being early users will help increase adoption within the farming community.

We validated that we can connect to the school through the USAid node. Although our contact at the school was on sick leave the principle made arrangements for another teacher to stay after school on Friday for almost 4 hours while we attempted to setup their node. We also noted the school now has a 15 seat computer lab.

The ACTS clinic staff were quite excited especially those that are volunteers from overseas. A number of ideas for the use of the network were put forward by the staff. These included using house bound AIDS patients to data capture for the clinic or simply to create new virtual relationships. Also suggested was using instant messaging as a form of anonymous counciling to allow school pupils to speak openly about AIDS. Issues noted

Support! We have already got our first support call from people who use operating system that know better then you what they want to do. This is a source of potential problems as people will phone the clinic and overbuden an already busy staff. We need to investigate how the community can provide that support or busget to pay for such support.

Lightning! This is going to be a serious problem, there are solutions such as mast spikes that we need to install.

Power downs. The clinic powers off most of their equipment at night thus taking the gateway down. We need to look at way to keep this up. The conection also require human intervention and we need to investigate automation of this taks.

Network separation. The mesh is currently on the clinic network. The clinic does deal with sensitive data but also for security and management issues we need to seperate the neworks from each other.

Usage monitoring, charging. We need to begin monitoring and controlling usage so that we can explore charging models in the future.

Windows... it puts junk on the network. We need to look at simple firewall rules on the nodes to drop NetBEUI junk from the network. Next steps

We will return to the site in a months time. At this time we wish to: separate the networks, install the school node, install a monitoring and access control system and announce a training course for the following month.

We will separate the networks by installing a new network card on the firewall. Most probably we will use NoCatAuth to control access to the network but need to investigate alternatives.

Other things to investigate are models for charging for network usage. Currently our strategy is to have farmers cross-subsidise the schools connectivity. A problem with this however is that other forms of connectivity are becoming relatively cheap (3G, etc) so this still needs to be reasonable.

February 19, 2005, Too much sun for a geek

Day 4: Had steak at the conservancy meeting. Talked a little about the mesh - nobody understood what I was tgalking about... But getting cheap Internet sounded cool to them. The farmers have serious problem with Telk^h^h^h^h connectivity, phones, lightning, macadamia thieves, water, etc.

Day 5: Dave J works for a big parastatal like company which means he always comes late cause someone wants him to do something (he tells me he was fighting Jhb traffic to get more Linksys boxes. Mainly spent the morning getting ntop (a great network monitoring tool working) and got EtherApe (nice graphical monitoring tool) working. These are quite good for seeing how badly Windows abuses network resources.

Spent the afternoon with David at Sikhile High. Was pretty fruitless we couldn't work out if it was the box, antennae, or sun that was messing us around.

Day 6: Installed the USAid node properly. Took a long time and got really sunburned. Note to self need a ladder, suntan lotion, umbrella and LOTS of water. Installed a 90deg patch panel. We had major problems getting links to the clinic which was frustrating.

February 17, 2005, More meshing

Day 3 cont...: Got USAid online, had been switched off not blown up by lightning. Phew. Installed a second mesh point which proved not to be needed. Worked out how to add parameters to the Kismet display, only worked out later that the signal strength bar graph is completely arbitrary. Will need to check all that again today. Installed an AP on the accomodation block, this seems a waste and we will probnably run cable from an AP installed on the staff flats instead.

Day 4: David J should be joining me today and we will be meeting with the farmers in the conservancy for a braai. I hope that we can raise the issue of what we are building and get people excited. Just before leaving yesterday I got everything up: I can ping all the other APs and can get out onto the net. I'll be checking this again remotely to see if I can get from USAid, to the clinic and out.

February 16, 2005, Meshing

I've had a busy few days here in Peables Valley:

Day 1: Setup a linksys on the roof of the clinic. Found that we really can't see Petra from here or anywhere actually, Petra is a nice high point but all buildings are hidden behind a large rock. We installed one at a house belonging to consultants for USAid. This is a good site as we can see the staff flats from there but none of the other buildings. The problem with this is we now need to find a good spot at the clinic so that we can see USAid without using two APs at the clinic.

I visited the school, unfortunately Mr Sihlabela is off ill until March. I met with the principal and two other teachers and have agreed to visit on Friday with David. The school can see USAid but we still need to confirm the signal strength.

In discussions with various people I realised that lightning is going to be a serious problem. With all the equipment on an AP costing about R2,000 it can cripple a network if they have a serious strike.

Day 2: A frustrating day trying to get Kismet working. This involved creating a patched kernel to allow the orinoco driver to go into Monitor mode. Finding, downloading and compiling takes a long time. I did manage to get onto the clinic roof and confirm that we couldn't see a signal from USAid. We did get a signal from the staff flats.

In discussing uses for the network we had some new ideas:

   * Getting house ridden patients a computer to get them communicating and getting outside virtually.
   * Having the same patient do data capturing to bolster his income
   * We also talked about using SMS and cellphones for compliance. Setting alarms for medicine taking. SMS messages that require a reply to check compliance. Notifying support group buddies about people not replying to SMS notification so that they can check that they're OK.

Day 3: Kismet is working! Not all the information is that useful for what I need to do. Met the clinic staff today to try and explain what we're doing. Today I'm checking signal strength from the school, its seems however that USAid is offline.

February 11, 2005, Equipment got

We now have the equipment for the FMFI Wireless project in Mpumalanga. We have 10 Linksys wireless AP which we're firmware upgrading. The cards are in a nice heavy duty all weather box. A picture would be great :). All the boxes are being moded to do power over ethernet which makes hte whole installation clean and simple. Currently it looks like costs are around R2,000 a node.

I'm lugging the lot down to White River this weekend and will erect 3 of the nodes. Mainly to test equipment and get a working initial network. The other nodes will be setup by the community, following the Onno Purbo style of network deployment.