Guest post by Mark Prosser, our network guru!

I’ve been a volunteer Network Consultant with DBIUA for nearly five years. As for how I got involved, and what being a remote volunteer from 4300 kilometers away is like… I’ll leave that for another post.

Of what I’ve achieved in the ~ five years, two things come to mind:

• Maintain & guide our monitoring and management strategy
• Overhaul our routing from static to dynamic

Although the first one is the bigger topic, since it’s something that consumes most of my cycles, I’d like to talk about the second point.

What was the routing like before?

DBIUA was founded by several crafty & innovative individuals. But they really went hard mode with their routing scheme in the beginning. Everything was static, everything was well planned. When I built my first Network, I would say the opposite was true.

Chris, our pioneer founder & Jefe Augustus, explains the old setup well in the below video from 2017

What’s the problem with this network?

Well, for starters, it was complex. That’s a lot of artificial hops between one PoP to another. But this was the design that serviced the many functions via a collapsed access PoP that DBIUA needed.

The network was also tedious… When you consider the amount of subnets displayed there, and remember that nothing was dynamic, well, that’s a lot of static routes to populate. We could mostly solve this with summarization. Each hub/access PoP would get subnets allocated from a greater /16 (10.X.x.x). Yet, this is still tedious as all blocks need to be programmed everywhere, to achieve a true full-mesh.

The second tedious element, is the operator intervention required. If a PoP had two potential ways to the upstream, an operator would have to switch the default route to the alternate path, should the primary path lose service. Floating static routes wouldn’t work as the local radio was our next hop, and this interface would not go down, should the upstream PoP go down entirely. IP-SLA could’ve been an option here (or BFD)… but wouldn’t that just be additional complexity for something that’s built into OSPF?

In Comes OSPF

Many grey beards will tell you, “OSPF is easy to deploy, but hard to master.” This is mainly because the default configuration of a flat single area comes with very little configuration that is greater than the sum of it’s parts; It provides keepalives, failover, and network state in 6 lines or less, per device.

The first step to making this work was to change the radios from participating in routing to doing what they do best, being bridges between routers (our EdgePoint routers). We would reconfigure one radio at a time and have the downstream interface on our router inherit it’s IP. This remain static at first, as the downstream radio still didn’t speak OSPF. But once we cutover the downstream radio, OSPF was alive, one PoP at a time.

When all networks were live, we finally had what we needed. It felt great to rip out all the static routing. All these routes still lived in the routing table, but for a lot less stress on the command line. It was also great to experience our first automatic failover via an automatic path.

So, what was missing?

OSPF’s defaults sometimes aren’t so smart. Let’s say we have a PoP with two potential paths to the internet, and these are equal cost. The problem is that, perhaps these links are not created equal. Because we need to thread the needle by pointing radios from home to home, some of these are pointed in not-so-ideal ways. For example, we have one path that is a low placed radio that shoots across the water. When the tide is low, this works great. But when the tide is high, it enters into the fresnel zone of the connection, thus causing packet loss and making the link less ideal.

The solution? Static costing. What we typically do is let links that are fine default to a cost of 10, or for links we truly prefer the bulk path of traffic we might set it to 1. But for links we want to be a backup path, well, we make them a cost of 30. This makes it so that only when the primary path goes down, OSPF will re-calculate and utilize the backup path as it’s main path to the upstream.  

We also have two upstreams to the internet, via our two providers. They are not equal in quality either. One is a not-so-reliable microwave backhaul to the mainland. Another we have a fiber connection we shoot our own 24Ghz radio to, and is much more reliable in quality and capacity. Thus, both routers have default-information originate set, but one of them we try to keep as a warm standby. Thus we utilize a failover setup via WAN Load-Balancing.

The other thing to consider with this method, is that you are applying a policy on traffic ingress to an interface. Thus you must set the traffic that is internal to your network to be exempt from this policy. We did this via these simple lines:

set firewall modify WAN_WLB rule 4 action modify
set firewall modify WAN_WLB rule 4 description pbr_tank_server
set firewall modify WAN_WLB rule 4 modify table main
set firewall modify WAN_WLB rule 4 source address 10.0.0.2/32
set firewall modify WAN_WLB rule 5 action accept
set firewall modify WAN_WLB rule 5 destination address 10.0.0.0/8
set firewall modify WAN_WLB rule 6 action accept
set firewall modify WAN_WLB rule 6 destination address 172.16.0.0/12
set firewall modify WAN_WLB rule 7 action accept
set firewall modify WAN_WLB rule 7 destination address 192.168.0.0/16
set firewall modify WAN_WLB rule 10 action modify
set firewall modify WAN_WLB rule 10 modify lb-group WLB
set interfaces ethernet eth0 firewall in modify WAN_WLB

note rule 10 which illustrates the policy is modifying ingress traffic 

If we didn’t do this, then traffic would be forced into routing table 101 , via the line action modify , which essentially is empty and vacant of our OSPF table. But when we say the destination is in RFC 1918 space, we do action accept which essentially means just pass the traffic as you normally would… utilizing the forwarding plane we have populated via OSPF.

Please tune in for part two where I go over the fundamentals of OSPF & how we configured it in detail.

See more of Marks blog at https://zealnetworks.ca/

It’s been quite a while since this site has been updated with new content, but that doesn’t mean that the DBIUA has closed up shop. We are just plodding along continuing to deliver internet connection to it’s members.

We continue to make adjustments to the network, both to software and hardware. On the software side we have switched over to mostly using Ubiquity UNMS to manage and monitor all our equipment (currently at 225 devices). For a short period of time we tried to use the CRM part of UNMS, but we had some billing issues with Stripe, and we rolled back to using the billing and financial software we were using before. On the hardware side we have been upgrading core backhaul radios to make sure there are as little bottlenecks as possible between members and our interent connection points. We have also added several redundant links so that if one radio goes down there is a secondary path to the internet. We also have maintained our upstream connections with both Startouch and Rockisland. If one or the other goes down, we automatically switch over to the other one.

People from all over the USA and the world continue to email from time to time asking for help or advice in setting up their own ISP like us, and I imagine there are a lot who just use what we have done and provided here to just get it done and stop waiting for some corporation to start caring and make their internet better.

The whole COVID pandemic made it abundantly clear how important a decent internet connection is for working and going to school remotely, and this brings me to the main subject of this post. Adding more members to the DBIUA.

A few years back, when Rockisland started to roll out their LTE wireless connections in the area, we stopped adding new members. You can read all about that here. But, truth be told, we have added new members since then, and the brutally honest criteria has been the following:

Are you in a location where one of our existing relay points can service you?

Creating a new relay point is a major undertaking, and sometimes it’s just not possible with the resources at our disposal. We have in the past placed relay points up in trees, but we are trying to avoid doing that because if something happens and we need to access that equipment, it’s a major time consuming PITA. Also, sometime the hills and tree cover in the area means getting a signal using the public low power frequencies we use are just not possible.

Are you not able to get a Rockisland LTE connection?

In some cases (due to the trees and hills), even the higher power LTE radios that Rockisland (actually T-Mobile) use can not reach certain locations, but we may have a relay point that can service you. But if you can use Rockisland, then you have an option. Adding more members does affect the speeds of our network for existing members, and it does cause more work for the volunteers running the system. Rockisland is larger company (supported financially by OPALCO), and so they have more resources to service people and provide a different level of service.

Do you understand that DBIUA is run by volunteers and that providing internet service is a complicated problem?

This last criteria really speaks to the partnership we have with our members. We try our hardest to keep the system running for everyone while keeping costs affordable and performance as fast as possible. If people are not able to understand that this is a two way partnership and are unreasonable with their expectations, either in asking for service or after getting service, then the partnership does not work. We have had to part ways with existing members due to these differences in views. I like to think about the golden rule. Treat others like you would like to be treated.

One last item, and this relates to membership transfers. We have had several people move away and sell their house. We will transfer the DBIUA internet connection to the new owners. It’s “contact free”. Meaning we just switch the billing over to the new owners. We have also had a few members move to a different part of Doe Bay. As long as we have a relay point in that new area, we will move your equipment to your new location.

So, that’s an update about the status of the DBIUA membership today, hopefully this clears up any confusion for those looking for internet options in the Doe Bay area of Orcas Island.

Over the summer we did some work at the top of the tank, secured some radios that were loose (oops), and while we were up there, we put a webcam up so we could enjoy the nice view.

Around Thanksgiving I started to take a snapshot every minute from the webcam and saved it. Then I stitched together them and made a little movie of the day. The webcam goes into infrared mode at night, which is pretty interesting.

Here is the first couple of days:

So, check out the Smalldognet Youtube channel to view things:

DBIUA Watertank playlist

Smalldognet Youtube Channel

It’s been a rough winter for the DBIUA.  Along with the normal power outages that happen from time to time, we also had some radio’s in some hard to reach areas go down.

Specifically one of the radio’s up at the top of the tree at my house.  This was one of the first relay points we installed, and is the one you see in the banner at the top of the site.  As usual, things end up breaking the second I got out of town on a business trip.  In this case, I was in Montana, and Chris Brems and Brett Marl had to schedule a tree climber to deal with the radio that was down.  The weather was also really crappy and there was not a great window to do the climb.

The connection that went down was the one that went from my house down to Nelson’s and there were a number of people downstream of that, so this was affecting a bunch of members.

As a fallback plan for replacing the radio in my tree, we also came up with a plan to put another powerbridge over at Patton’s that pointed over to Blakely just in case.  And Mark Prosser was able to modify the network so that we could service those people via this connection as a backup.

Well, it was a good thing we had a plan B, because when we went up the tree, it turned out that the little tab on the cat5 connection broke, and there was not way to re-mod the end up in the tree.  So, they came back down the tree.  But before they did, they noticed that the top part of the steel bar was not connected to the tree any longer.  This little piece of information probably got lost in translation.

So, those down at Nelson’s and downstream limped along for a while via the new Patton link while we waited for the weather to calm down.

Then a few weeks later we got an alert that the connection to Hickey went down.  I wrote this off as maybe being a power outage there.  But, around the same I noticed that my connection back to the tank was not that great, which was sort of strange.

About a week after this we scheduled the tree climbers (Ian and Pheonix) to come back.  By this point I had purchase all new radios for the top of the tree.  We switched out the old rockets for new Prism Stations, and an Iso Station to point back to the tank.  On the tank side, we also upgraded that old rocket to a newer Rocket AC.

Before they climbed up the tree I decided to fly my racing drone up to check out the radio’s just for fun.  And, guess what I found.  The whole radio group had fallen over and was laying on it’s side.  Check out the video below.

That would explain why my connection was bad, the radio was 90 degrees from where it should have been, and the other radio that should have been connecting to Hickey was pointing up to the moon!

After about an hour, we had all the old radio’s down, and the new radio’s back up in the tree, and everything connected back up again.  Here is another video of the drone flyby of the new setup.

Yesterday, December 14th, 2017 was a sad day for the internet, as the FCC killed Net Neutrality.  The biggest loser in this are individuals, followed by smaller groups of individuals, and small companies.  The winners are the big corporations, and the larger they are, the bigger they win.

How will this affect the DBIUA, as a member owned ISP?

The good thing is we are already a small group of people who have pooled our resources.  We have over 60 members, and we do everything we can to provide each of our members the fastest connection possible to their home for $35/month.  Depending on where you are in our network, this may range from 5mbps up/down all the way up to 50mbps up/down.  But, we don’t specifically throttle or prioritize traffic in our network.

The other good thing is we have two different network providers now.  Startouch Broadband, and Rockisland Communications.  We have our network gateways setup to split traffic between them.

Luckily both of these providers were willing to provide us a wholesale connection.  But, depending on where you live, and what your options are, even if you can put together a group of people who want to share a wholesale internet connection, you may not be able to find anyone who will provide you that connection as it will be more profitable to deal with individuals directly.

We pay Rockisland about $900/month for our connection that 60 different individuals use.  Rockisland’s normal price for individuals is around $75/month.  If these 60 individuals where direct customers, that would be $4,500/month for them.  The financial incentive for all these ISP’s is NOT to sell wholesale connections, but deal with individuals directly.  Also, when dealing with individuals directly, they can upcharge for better service for those individuals who want it.

Even though we have two network providers, that is not the end of the story, because each of them have to connect to a bigger fish upstream.  Their upstream providers could decide to charge them differently and throttle/prioritize traffic, and then the right thing to do from a business perspective is to pass these costs back onto their customers.  And then we would have to pass that back to our individual members.

The part of this that really bugs me, is the that the organization that should be looking out for the individuals in this country, and making sure they are not getting screwed is our government.  It exists to represent and protect each of us, and it just screwed us all.

I setup a test network in my garage last weekend, so we could test out some network changes we are going to make.  This was also a great opportunity to try and give those of you interested some more technical details about how we have things setup.

Here is a photo of the setup now on the wall of my garage.  This replicates just about all the different working parts of our network.

And here is a youtube video explaining things.

The DBIUA has been up and running for about 3 years now, and way back when we decided to use Ubiquity gear in our network for a various reasons, mainly their AirMax M5, M2, and M900 stuff.  We also used the 5 port toughswitch in all our relay point locations.

As time has gone on, Ubiquity has continued to evolve their product line.  And my new favorite product is the EdgePoint Router, specifically the EP-R6.  It is just a little larger than the toughswitch we are using in our relay boxes, but it’s so much more capable than the toughswitch, and will allow us to rework our network to make it much easier to manage.

We are in process of switching out (ha, ha) all the toughswitches with EdgePoint Routers over the summer and doing some work to simplify our network so we don’t have to program manual routes in all our radio’s.

As part of this change, I’m setting up a test network in my garage to be able to try out the different settings we will need to modify, so we can make sure we don’t accidentally brick a radio that is up in a tree when we do the real updates.

Once I get this network up and running, I will post again here showing the test network, which will give you an idea of all the pieces and parts you need to build out your own network.

In August, the DBIUA board decided to stop accepting new members, and those members who were still on the waiting list, we decided to refund their membership fee and not connect them to the network.

We came to this decision for a couple of reasons.

First, while we have been working on our redundant connection with Rockisland, we learned that they are putting in one of their LTE poles over on Blakely.

You may have seen these around in other locations, like at the Obstruction Pass power station, by their offices in Eastsound, over by the transfer station, and at other locations around the county.

These LTE poles are providing T-Mobile cell service as well as providing fixed LTE wireless internet from Rockisland.

This new pole on Blakely should be able to service all of the people on our waiting list.

The other reason we have decided not to expand our membership is our volunteer labor (mostly me- Chris Sutton), does not have enough time to deal with the existing membership and network, let alone install new members.

This doesn’t mean we will be shutting things down and closing up shop. It just means we will be spending our time keeping the current system running.

One thing I have not touched on yet is what we use for billing our members, accounting, etc.

My day job is doing software development (http://smalldognet.com), and one of the products I created is software to run community foundations.  This includes a full blown online fund accounting system, credit card processing, customers, donors, AR, AP, checks, etc, etc.

So, I already had software we could use to keep the books, all I needed to add was an “ISP” module that tracked the different radio’s, determined if a radio was backbone equipment, or member equipment.

The DBIUA Network module keeps track of everything, helps with the programming of the radio’s, and also feeds the nagios monitoring system.  When we add someone new to the DBIUA Network module, this automatically gets pushed out to nagios.

This system tracks the location of everything as well (latitude/longitude), and so we can also spit out a physical map of the network and how everything is connected.

The best part of this is really the automatic billing each month.  My software is already setup to integrate with Stripe payment processing (http://stripe.com), and so we have a page where members can login, and give us a credit card.  This is not saved in our system, but instead is saved over at Stripe.  At the first of each month, I press a button that automatically creates invoices for everyone.  Then another button charges everyones credit card using the data saved at Stripe.

I’m sure some of you are thinking, this would be really cool to use for your own WISP.  Maybe you are using Quickbooks or something and your monthly billing is a total PITA.  For now, you will have to continue to suffer, but maybe in the future I will figure out how to make this available to others as well, but right now it is very DBIUA centric.