In my prior post, I talked about troubleshooting power issues.  And this past week we had another problem with out system that turned out to be power related again.

At our relay points, one of the difficulties is figuring out where the closest AC power is, and how to get from there to where we need our radios to live.  Generally this distance is not very far.  But in a couple cases it’s a long ways, like with the relay point in the middle of Tom’s field.  In this case we trenched the 120v AC power all the way to the relay point.

In another case over at Jim Nelson’s point, we had to run 200 feet from the closest house, out to a tree on a point, and instead of bringing 120v AC out to the tree, we ran the power over POE the 200′.

At first we only had 2 radios at this relay point, and everything worked really well.  Eventually we added a 3rd radio, and most recently we added a webcam.  Adding the webcam pushed everything over the edge and suddenly everything at that relay point was rebooting over and over and over.  WTF?

Well, there is this thing call line loss, when transmitting power over certain distances.  Here is a great little webpage and helps you with those calculations.

http://www.calculator.net/voltage-drop-calculator.html

So, we are transmissiong 24v DC power, 200′, and we are using Ubiquity Carrier Class ethernet cable, which has 24 AWG wire.  And POE uses 2 pairs to carry the power.

The last thing we need to fill in is amps.  When we had 2 radios, each using 8 watts, that means about 0.4 amps (8 watts/24 volts) per radio.

Plugging all this into our calculator shows we end up with just short of 20 volts at our radios.

Adding the 3rd radio (and amps up to 1.2), we fall to just shy of 18 volts, and adding the webcam we are under 16 volts, at which point things obviously started failing, probably the touchswitch that all this was plugged into.

So, our solution was to turn off the webcam to get everything running again, and then to order a bunch of 12/2 outdoor landscape lighting cable, and use that to bring the 24v power out to the relay point.  Plugging in 12 AWG wire into our calculator across 200′ and 2.5 amps (the max the power supply will put out), gives us 22 volts at the relay point.

So, if you are running more then a couple of radios across a long distance POE link, you really need to do something different for power because that tiny 24 AWG wire just doesn’t cut it for high power needs.

Computers and the networks they us to communicate with each other are complex things, and from time to time, things stop working, or don’t work as well as they used to, and you have to figure out why.  Troubleshooting is a bit of an art, and in this post, I will go over various troubleshooting stories, and how to try and avoid rebuilding things from scratch when you just forgot to check a checkbox on some configuration screen.

The most important part of troubleshooting is knowing there is a problem in the first place, and having as much information as possible to help figure out what is wrong. You need a system that is checking your entire network and can alert you when something goes wrong. We have installed Nagios, which is an open source network watchdog program. This checks all our backbone radios and routers every 5 minutes. Individual member endpoints are checked every 15 minutes. So, if a radio goes down, Nagios will alert us to this fact when it happens.

When you are alerted to a problem, the next useful piece of information is to be able to see what may have been happening in the past. And for this we have another system installed called Cacti, which is another open source data logging program. This system checks every router, radio and member router every 5 minutes, and logs how long it takes to contact (ping time), how long the system has been up (uptime), if it’s a radio, it records the signal and noise values as well as raw bitrate. It also records how much data has been transfered recently (bandwidth). All of these metrics are very useful in troubleshooting any problems. And without these two systems, we would really be working in the dark when something went wrong.

Start with the basics

When we see a radio go down, the first thing we first try and confirm if there is power to the radio. Lack of power is way more common than many people realize.

We have battery backups on all our backbone radios, so sometimes the power loss happened several hours before. One case is the infamous sheep. When we were first building things out and installing one of our relay points in the middle of Tom’s field, we had a temporary extension cord running out into the field to power our relay point. In the middle of the night we got an alert that “tillman-pb-5-a” was down. In the morning we went out to the field, and found that the extension cord was unplugged, and apparently a sheep had scratched up against it during the day and unplugged it, and so then about 8 hours later, the battery died, and the radio went offline.

Another time, we got an alert that “shipstad-ar” was down. This is a member wifi router, and I knew that this was probably not on a battery backup. The rest of the equipment at the Shipstad’s was still up, but my gut said it was now on battery backup. In calling to the Shipstad’s house, I found out that they were doing something in the garage and had popped a breaker turning on a heater or something. This had turned off the wifi router, and this had also stopped sending POE power to the rest of the relay point equipment at that location. It is this type of event where we are working on being able to monitor grid power at our relay points, so we know if there is a power problem and we are on battery backup.

Sometimes the power problems are not lack of power, but not enough (meaning not enough amps).

Early on in building out our network, I put 3 radios up in one of my trees, two nanostations, and one rocket. To make life easy, I ran one wire up the tree. Nanostations have 2 network plugs and allow you to daisy chain another device. Down on the ground, I tested running one POE cable into a nanostation, then on the secondary port, running another cable to the 2nd nanostation, then from the secondary port on that radio, into the rocket. Everything turned on and lit up and I was able to login to each radio on the one wire.

But, when it all went up in the tree, and we started running traffic over the radios, everything started rebooting. Well, turns out you can only daisy chain once, and when the radio started sending traffic it started pulling more amps than was available, and so things rebooted. So, the lesson on that was to run one wire for each radio up the tree.

But, there was another location where we only needed to have 2 radios in the tree and this worked well, even under load. We needed to add a 3rd radio in that location, which meant we needed to install a touchswitch, instead of running the 2 radios directly from the Tycon Power charge controller directly. So, we switched the 2 radios to one port on the toughswitch, and the 3rd radio to another toughswitch port. Then we started to get random alerts that the 2 radios where going down. Turns out you can’t run a nanostation chained to another radio off a single toughswitch port. So, we ran a dedicated wire to each radio and each had their own port on the toughswitch.

After you have made sure there is power to a location, you next should check that there is not a problem with the physical wire that carries the power to the radio.

We had one time when a radio went offline, and what happened was a small branch came down, and must have hit the ethernet cable, and the cable was not completely “clicked” into the network port on the back of the radio, and so the cable popped out.

There were several times when even though the radio had power lights and was connecting upstream, downstream on the network packets were not flowing. This was usually a problem with the crimping of the cat5 end. I had one of these recently, and even though I have done hundreds of these end crimping connections, every once in a while I’m not paying attention, it’s a little dark, or i’m taking to someone, and one wire goes in the wrong place. Then things either don’t work, or they half work.

One time we had a faulty POE power brick, where the little wires in the brick that are sprung to connect tightly to the cable end, where stuck, and only completed the connection if you pushed the cable end into the POE brick really hard. This was the intermittent power problem.

Then we just had another case where a member had a faulty powerstrip, that if you touched it wrong, it would turn off the power.

After confirming all the power related issues, we then get to the programming of the radios….which will be another post because this one has gotten really long.

One of the tricky parts of creating the dbiua network was getting the relay points setup, with a small POE switch, and battery backup.

We initially used Tycon Power Systems UPS-DC1224-9 system.  This was a great little box that had a 9AH battery and a charge controller, and just enough room to get a Ubiquity Toughswitch5 in there.

But the 4 allenhead screws were not the best and after being out in the weather for a while, they would break.  So, we switched things out and used all the same parts (9AH battery, TP-SCPOE-1214 charge controller, and toughswitch), but in a Hana Wireless NEMA 12x10x5 fiberglass box.  But, the biggest problem was mounting all the stuff in the Hana box efficiently.

What I ended up doing was designing a bracket to hold the battery up at the top, and then 2 brackets to hold the toughswitch above the charge controller.  I 3D printed both of these things (the blue things in the picture below).

I have uploaded the design of both of these brackets to Shapeways, so if you want them for your own project and do not have a 3d printer, you can just order them from Shapeways.

https://www.shapeways.com/shops/dbiua

You will need 2 toughswitch9 and 1 batterybox12.  The prices for both of these are not marked up at all.

One last part that you need is a POE splitter (Tycon Power Systems POE-SPLT-S
Passive Splitter 5.5/2.1mm DC Shielded): https://www.streakwave.com/itemdesc.asp?ic=POE-SPLT-S&eq=&Tp=&o1=0

This allows you to pull the power off the charge controller POE out port, and power the toughswitch.  In the picture above you can see one end of this splitter plugged into the first port of the switch.  The rest of the splitter is under the switch.

Our latest version of this equipment box includes a RaspberryPI (designed by Brett Marl), that monitors the power of the battery, as well as doing internal speed tests on our network.

In a previous post I mentioned the wifi routers we install in members homes.  These are the last part of the network.  So what about everything else?

We use a variety of radios depending on the situation.  For PTP links we generally use PowerbridgeM5 on both ends, though sometimes we have a PowerbridgeM5 on the upstream side, and a NanoStationM5 on the downstream side.

For one PTP link from the water tank, we have a RocketM5 with 30dBi 2′ dish, and a NanoStationM5 on the other end.  The NanoStationM5 is up at the top of a tree, and sways around a bit.

For our PTMP (Point to Multipoint) links, we always have a RocketM5 with either a 16 or 19 dBi sector, or sometimes a 13dBi omni antenna.  On the downstream side we use either NanoStationM5, or NanoLocoM5, or NanoBeamM5-16, or sometimes for a long link, a NanoBridgeM5-22.  The NanoBeamM5-16 are really great little radios as they have a nice integrated mounting bracket.

Some of our links use 900mhz, and for these we will use a RocketM900, with a 13dBi sector (what I call the water heater, because the thing is huge).  Then on the downstream side we have NanoLocoM900.  Sometimes we will use a NanoLocoM900 on both sides of the link if the link is not going very far, or there is just one client radio downstream.

In one location we are using 3.65 Ghz (which requires an FCC license).  Here we have a RocketM365 with a 12dBi Omni, and then NanoStationM365 for downstream radios.

And in another location we are using 2.4Ghz, and a RocketM2 with a 16dBi sector and NanoStationM2 downstream.

These are all part of Ubiquity’s AirMax line.  We are not using an AC radios yet, but we might switch to some of these on our backhaul links in the future.

Our standard member install usually (always), has one radio outside the house.  This is some form of their airmax line.  We sometimes also provide a 2.4ghz wifi router for the member inside the house.  This is not always required if they already have a wifi router.

In order to make things easy, we decided to standardize on a Ubiquity product here as well.

In the very beginning, we used their basic AirRouter.

We then started using the AirRouter-HP, which has a better range, and is powered by POE.  it is a little more expensive.

Then I ran across their AirGateway, which is this tiny little thing that plugs directly into the POE brick.  They also had an AirGateway-LR, which was similar to the AirRouter-HP.

After doing some installs with these, I decided I liked the AirRouters better, specifically the AirRouter-HP.  Why?  Not totally sure, but they seem more tried and true, and you have the ability to plug more than one thing into them if needed.

Probably all Internet services give people a choice of either speed or data caps.

The DBIUA decided to see what would happen without imposing either of these on its members.

We got the fastest upstream connection that we could, and then we built our wireless network out and tried to provide the fastest speeds possible in an affordable manner.

In order to test the speeds on our internal network, we installed a speed test mini webpage on a server at the water tank.  This allows people to test speeds to the tank.

If you are one or two hops from the tank, you can probably get upload and download speeds in the 40+mpbs range.  If you are several hops away, then that tends to drop to the 20’s.  And some places it’s around 10.  But, that’s a far cry from the 1.5mbps that you sometimes got with DSL.

So, what does our overall bandwidth usage look like going out of the tank to the internet?

This is a snapshot of 2 days.  Notice the spike in the evenings.  So even though people may be able to download faster, the reality is they don’t consume that much, and it’s only generally in the evening.  Here is another graph that shows this same traffic over a weeks time.

So, what about individual usage?  Not all ISP’s graph this data, but we decided to do this so we could manage the network and identify any issues.  I think those ISP’s that might do this, would probably not share this information, because it shows we actually use WAY less than we think we do.

Here is the usage graph for the connection from my house for that same 2 day period:

Notice the scale on the left.  3.0.  Not 30.0 like the above graph.  And the usage is way less.  The larger blob at the far right is watching some video.  The little spikes are various downloads.  General web browsing, or youtube at lower rez is the other green blips.

Here is another person, with a different usage pattern:

Notice the scale change again.  This is someone streaming something high def (the large green blobs).  But, there are still large swaths of time when nothing is happening.

Here are a few more

All of these individual usage patterns flow together to create the one at the top.  We have not had to throttle anyone, or impose data caps.  We allow everyone to use what is available at any given time on the network.

There have been times when a lot of people were streaming something around the same time, and guess what happened?  Things slowed down a little bit for everyone.  Sometimes there may have been a little buffering, but in general it has not been a problem.

The million dollar question is how much speed do you really need?

In my opinion, if you have a reliable 2-3mbps available to you, that is plenty.  If you can burst to faster speeds as needed, that is an extra bonus.  6-8mbps means you can stream very high def video.  But using 50 or 100mbps for long periods of time is actually not very common.

And, personally I have had times when slow speeds are not on our end, but instead on the other end of the connection, at the data center side, where a webserver might be throttled, or on a slow connection.

So, hopefully this gives you all some good real world information about our little socialist network experiment 🙂

When we launched the DBIUA, in order to cover our upstream bandwidth costs, and to pay down our 3 year loan for our capital equipment, we needed 23 people to commit to paying $75/month.  We got those “early adopters” as Chris Brems likes to say, and then over the past year+ we have slowly added more and more members.

Today, we have 61 connections, with 10 on the waiting list to be installed.

We have almost 3 times the number of members needed to pay for running the system.  What are we doing with all that money?

We had to spend some money on backbone infrastructure to service more people.  And we have installed a backup link at the water tank, so our upstream bandwidth costs have increased a bit.

We purchased backup equipment to have on the shelf in case something bad happens (like a tree falls down).

We are paying off our 3 year loan a year early.

And….we are lowering our monthly rates.  Just like we said we would.

After talking through some ideas, we (the board) came up with the following, which we thought was fair to those who have been with us from the start.

Your first 18 months on the system costs $75/month.  Your next 6 months are $65/month, then 6 months at $55, then after that $45/month.

Our first round of members will start seeing their bills go down in the next month or so.

Even with this tiered pricing model, our projections still say we will end up with a lot of cash in the bank, so we are looking at other capital credit payments to members, along with continued investment in our network.

Ever since moving back to Orcas several years ago I have been looking for ways to connect to the internet.

The good news is today, in 2016, there are more ways than ever.  But making sense of them all is difficult if you are not a tech geekhead (like me).

This post tries to shed light on the different options available to those on Orcas.  I will try my hardest to make this easy to understand.

As with many other decisions in life, we weight our choices based on things like features and price, and we need to be able to see through all the marketing doubletalk.  Internet connections are no different.

CenturyLink DSL

Even though you can get DSL through others (Rockisland/Orcas Online), it’s CenturyLink who is providing internet over the copper wire that you use for your phone.

Depending on where you live and how far you are from the local “remote”, this might be an option for you.  It is probably what most of you already use for internet.

The remote is either the main office in Eastsound, or a box that is on the side of the road somewhere.  Usually it’s within a mile or two (or three+!) from you.

If you are close enough to the remote, and it is connected to the rest of the world by a fiber optic cable, then you can probably get speeds of 6mbps or more download, and about 1/10 of that upload (see this page for an explanation of upload/download speeds).  Otherwise you will only get about 1.5mbps download and about 1/10 of that upload.

In some locations even if you are close enough, CenturyLink will still not provide you access because too many people are already connected to that remote.  Also if there are too many people connected to a remote, speeds for everyone suffer.  This is called “oversubscribing”.

The good part of DSL is it’s cheap, and by bundling with your phone service, you probably only pay $10/month.  The down side is sometimes it is not very reliable, and dealing with a huge company like CenturyLink when it doesn’t work can cause you to jump off a cliff!

Satellite

Many people on Orcas have a satellite dish on their house so they can watch TV.  So, it seems like you should be able to get internet this was as well.

You can, but like with DSL there are some good and bad issues.  The good is that the speeds are better than DSL.  You still have slower upload than download though.

The bad is the cost is more than DSL.  Also, because your signal goes from your house, out into space, and then back down to earth again, this thing called “latency” comes into play, and even though the speed is good, it takes a little bit before the speed starts happening.  For example, you might click on a link on a webpage, and there will be this 2 or 3 (or more) second lag before the page comes back.  But once it comes back it all downloads quickly.

One of the bigger downsides are usually “data caps”.  This means if you consume too much of the internet, you either get cut off, or the speed goes down to really slow, or you get a huge bill at the end of the month.

Things that consume a lot of your internet connections are watching video, backing up your computer, uploading images to places like Facebook.

You might think that a data cap of 5GB (gigabytes) is a lot.  But watching just one Netflix movie will probably blow through that.

Verizon/T-Mobile/AT&T

You might already have access to the internet over your mobile phone, and depending on where you are, you might have “LTE” coverage, and get speeds that are better than what DSL or Satellite can provide.

Also, some of these companies can provide you “Fixed Wireless LTE”, which is a little box that lives at your house that connects to the internet the same way your phone does.

Both of these options have the dreaded “data cap” like with Satellite, and either they will cut down your speed to a crawl when you use up your data, or they will cut you off, or they will send you a big bill.

Again, don’t let large data caps like 100GB fool you, you WILL consume it all, and you will pay them a LOT of money each month.

Mount Baker Cable

I don’t have much experience with this, but from what I know it’s not available everywhere, the speeds are ok, but not always consistent, the reliability is like DSL, and I think there might be data caps as well on this service.

Wireless via Orcas Online or DBIUA

Orcas Online and DBIUA both provide access using small radio’s that run over public radio frequencies.  These radio’s can provide speeds up to 50mbps (or more) both upload and download across long distances (10+ miles in some cases).  These radios do require good line of sight, which means you can’t shoot through a bunch of trees.

The good parts of this option are the price is affordable, installation is fairly quick and easy, and speeds are better than DSL.

The downside is that because these radios use public frequencies, at times there can be interference that causes reliability issues.

Prices for Orcas Online vary depending on how fast you want to go.  DBIUA is a flat fee, and provides whatever speeds it can to you, and the overall speed of the system is shared with everyone (this is a very uncommon model).

This wireless option is a shared system, so that if several people are connecting to the same upstream radio, they all share the available speed.  So not everyone can get 50mbps all at the same time.

Access to this type of connection depends on where you are located.  DBIUA is obviously only on the Doe Bay Area, but Orcas Online has links and relay points all around Orcas and on other islands as well.  You just need to be able to see the correct access point.

Startouch Broadband

Startouch is based in Bellingham and provides commercial microwave connections across the state.  These are licensed FCC links and are expensive to install ($10,000+), but there is no interference issues. They are also capable of very fast speeds. The Orcas public school and library both have these types of connection from Startouch. Notice the large dish on the roof of the library.

The DBIUA also uses one of these links for its upstream connection to the Internet.

Startouch also provides wireless business service at slower speeds and lower costs using radios like Orcas Online and DBIUA.

The requirement to use Startouch is being able to see one of their towers which means Mt. Constitution or towers on the mainland.

OPALCO/Rockisland

Before Rockisland was purchased by OPALCO, they provided similar wireless services that Orcas Online and DBIUA provide.  But after being purchased by OPALCO, they seem to not be offering these services.  The push now is to bring fiber to your home, or at least those that want to pay for it.

Internet over fiber is very fast and reliable.  Speeds are the same both upload and download.  But it’s very expensive to get to you.  Even if you are less than 100′ away from the backbone it’s still going to cost over $1,000 to hookup.  And it takes a LOT of time and effort to get installed because they need to trench all the way to your house.

The other option they are selling is “Fixed Wireless LTE” like the mobile phone providers.  Except with no data caps!  The monthly cost is about the same a fiber, and the speeds are similar to other wireless options.  Installation cost is very affordable ($0?).

This is available now in areas where they have put up their tall poles with cell equipment on them, like at the Eastsound office, and at the Olga substation.  Their plan is to put these poles all over the county to improve communication with their linemen, as well as provide internet to people.  They have also partnered with T-Mobile to allow them access to this equipment for mobile phone service.

Rockisland’s message is that this wireless option is available for areas where fiber is hard to reach or very expensive.  But in my opinion there is no technical reason why everyone should not be able to get this and be forced to pay for fiber.

Summary

There are many different options available today for internet access on Orcas.  In some locations right now you may only have one available to you.

But as OPALCO builds out their LTE Wireless network, and as Orcas Online continues to expand their network, and others learn how to reproduced the DBIUA model we will have even more options in the future.

Startouch is working on our main radio link doing some config changes remotely.

This was a good test to see if I had our cisco router setup correctly to fail over to the backup link, when the primary link went down.

It didn’t work.  So obviously I don’t have this setup correctly.  I thought the magic was in the routing area:

ip route 0.0.0.0 0.0.0.0 1.2.3.4
ip route 0.0.0.0 0.0.0.0 5.6.7.8 20

1.2.3.4 is the primary link, and 5.6.7.8 is the backup, and the “20” after the backup I thought said to use this if the primary went down.

But, this didn’t work.  Looking into why, but for now, just needed to go into the router, do a config terminal, then swap those 2 lines, and put the “20” on the 1.2.3.4 and blank out the 5.6.7.8.

Early on in the DBIUA we didn’t have much money, and had to be pretty frugal in what equipment we used.

Someone else told me early on, pick a radio vendor and stay with them.  Don’t mix and match.  We decided early on to use Ubiquity radios.  They had a wide range of options in their AirMax line, and the prices were very affordable.

To this date, I don’t regret making this choice, the radio’s have worked extremely well, and have been easy to setup.

We also decided to stick with the Ubiquity brand for other network equipment, including their 5 port tough switch, as well as their Edge Router and Edge Switch.

The 5 port tough switch is great, and we use these at all our remote relay stations.

The Edge Router though we had some problems with.  A few months back we started noticing that large downloads would just die mid download due to “packet errors”.  Sometimes if you tried again it would go through fine.

Doing a little googling it seems that others were having problems when they were going through an Edge Router.  I made sure we were updated to the latest firmware, and then I did some tests where I tried the download behind the Router, and then again in front of the router.

Behind the router I could get the download to fail every time.  In front of the router worked no problem.  So, it wasn’t a problem upstream.

So, the solution (now that we had some more money in the bank), was to upgrade to a Cisco 1900 router.

This was not nearly as plug and play to setup as the Edge Router was, and required some serious googling around to get NAT working correctly, and the failover to work across our primary and backup link, but eventually I got it configured correctly and we were back in business with no errors on large downloads.

Plus, as one of our members Brett Marl says “will be much more bomber with cisco hardware in the shed”.

For those who are interested, here is our cisco config, with NAT setup for our 10.0.0.0 network, and failover to our backup link.

interface GigabitEthernet0/0
 description primary link isp1
 ip address aaa.aaa.aaa.aaa 255.255.255.248
 ip nat outside
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface GigabitEthernet0/1
 description secondary link isp2
 ip address bbb.bbb.bbb.bbb 255.255.255.252
 ip nat outside
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface GigabitEthernet0/0/0
 no ip address
!
interface GigabitEthernet0/0/1
 no ip address
!
interface GigabitEthernet0/0/2
 no ip address
!
interface GigabitEthernet0/0/3
 no ip address
!
interface Vlan1
 description inside LAN segment
 ip address 10.0.0.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly in
!
ip nat inside source route-map isp1 interface GigabitEthernet0/0 overload
ip nat inside source route-map isp2 interface GigabitEthernet0/1 overload
ip route 0.0.0.0 0.0.0.0 aaa.aaa.aaa.aaa
ip route 0.0.0.0 0.0.0.0 bbb.bbb.bbb.bbb 10
!
!
route-map isp2 permit 10
 match ip address 100
 match interface GigabitEthernet0/1
!
route-map isp1 permit 10
 match ip address 100
 match interface GigabitEthernet0/0