More Pi Backup

So a million years ago or so I wrote about the Pi backup server. ( Actually, I had two. Originally each was a Pi B running Pidora, with a pair of Passport USB drives.

At some point I added a second RAID pair of WD Red drives to one Pi, and a Pair of pretty ancient Seagate drives to the other. Eventually I replaced the Seagates with another pair of WD Reds.  Somewhat more recently I upgraded the Pi B's to Pi 3's, one running Fedora 27 and the other CentOS 7. The CentOS Pi is doing nightly backups, the F27 Pi is my NFS server.

Every so often one of the drives drops from it's RAID pair. Sometimes it is simply a matter of re-adding it, sometimes power on the drive needs to be recycled. This behavior didn't change going from the Pi B to the Pi 3. But I have been logging these events, and going back through the logs, it is always the Passport drives.

I had chosen the WD Red drives because this is kind of an NAS application. There is some other color for RAID, but the Red drives have worked well.

I suppose the lesson is not to use USB drives in 24x7 service. Drives made for the purpose work a lot better. I also continue to feel obligated to pitch the idea of a Pi with a RAID array for backup. It's a pretty cheap alternative, it just sits there and mostly gets ignored, but everything is backed up with no pain.

To Do

OK, there are endless to-do list applications, each with its own plusses and deltas.  I tried the emacs todo without a lot of joy.  I more or less settled on using bugzilla since it allowed me to not only capture relationships and estimates, but also to keep notes on various projects or bits of projects.  I tend to have lots of things that I need to do "right now", but even more things that I would like to do if I ever get one of them round tuit thingies.  BZ works well for this in terms of capturing things, and especially capturing thoughts on those things I want to get around to some day.

The problem with BZ is that I have to go to it to see the list. Even then, the default reports ignore deadlines, focusing only on Priority and Severity.  I currently have over 100 active "bugs", most of which I hope to eventually get to, but most of which I would also rather not have distracting me at the moment.


I made a script to print a report of the top few upcoming deadlines and put it on the crontab every few days, but I quickly tended to ignore it.

So I started gravitating toward devtodo, although all the "important" stuff was duplicated in bugzilla.  One of the really nice features of devtodo is that each directory can have its own todo database, and they can all be linked to higher level databases.

I tend to be a command line sort of a guy, and it is really, really nice to have the list of things to do on a particular topic right there when I'm working on that particular item.

The downside is that devtodo really doesn't have any sort of deadline concept, and there is no way to reasonably keep details of a to do item.

So I finally broke down and made a PHP script to read the bugzilla database and populate the devtodo databases each morning on the crontab.

It is nice to go to a directory and see only those things relevant to that particular area, and when I descend to a lower directory, see the related subset (see the above image).  If I need to be reminded of some details, I include the bug number so I can quickly get to it in bugzilla.

I really should put this in gitorious so others can easily steal it, but it is highly specific to my bugzilla customizations, my personal directory structure, etc.  So sorry, you can't just grab the code easily.  I tend to enjoy generalizing things, but honestly, it will be a while before I find the cycles to do that.

But it is pretty simple stuff.  Read the categories and/or components tables to get descriptions, rummage through the bugs table to find the high priority items, and write out the xml.  If you are of the command line persuasion like I am, you might just find it satisfying.

A Raspberry Pi Backup Server

I have a fair number of boxes on my LAN, most pretty old.  From time to time, of course, I have a failure, and a failure is always a disaster.  Most recently I had two systems fail, an ancient FC4 (!)  box, and a box that contained my only actual Windows system, although more often it was booted to Fedora and used pretty much only for IRC.

Since it is awfully handy to have IRC on a different box/keyboard/display than the one I am working on, for a while I put my IRC on a Raspberry Pi while I sorted how best to repair/replace the aged Windows box.

That made me think that perhaps I could move other functions to the Pi.  If I had a "Pi Farm" with a Pi dedicated to each function, then a failure wouldn't be such a big deal.  By default, Pi storage is SD, and that isn't so good for stuff that gets written a lot, so external storage is a must.  But that leads me to the realization that if the external storage was RAID 1, a failure would now be a minor inconvenience.

If you think about it, it is pretty simple business to save off an image of the SD card so that recovering from an SD card failure is simply a matter of writing another SD card.  A failed computer is a $35 card replacement.  A failed hard drive, simply replace the drive and let RAID do its thing.  So now, most failures are changed from a disaster to an annoyance.

So step 1 was to see how well RAID works on a Pi with external drives.  I picked up a couple of 2Tb USB drives and made a RAID 1 array on a bare Pi (Pidora 20).  Then I made a raft of cron jobs to rsync any important directories to the Pi.  Turns out that all of my active directories together only amount to a half a terabyte.  Yes, it is a little doggy syncing large directories to the Pi, but it is all happening behind my back so I don't see it.

I don't do any operating system directories or directories that already hold backup savesets, so at this point, a failure is still a big deal, but at least I am confident that I won't loose any data.  And recovering a file from an rsync mirror is a whale of a lot faster than fishing through a bzipped tar (although I don't have multiple generations saved on the Pi).

In any case, this works quite well.  So time to lay out just what I would like my Pi farm to look like.

Problem is, each Pi needs storage, and while USB drives aren't horribly expensive, when they start to multiply it adds up.  Worse, most of my important functions don't need a lot of storage, but external drives smaller than half a terabyte just aren't available, and a terabyte costs barely more than a half.  Using a thousand gigs for an application that requires one seems like horrible overkill.

But if my storage is RAID, then the risk of consolidating the storage onto one server is minimal.  Yes, a failure of the server takes out multiple services, but simply replacing a Pi is pretty simple (and cheap) business.  Replacing a failed drive is a little more expensive, but the RAID catchup will take care of the hard work.

So the plan now is to have a second RAID server, this one for active services.  Then a Pi for each significant service.


Over time I may find that more services (and hence more Pis) are needed, but there is plenty of capacity in the storage server.

A shameless hack

Back in the olden days, when I was afflicted with Windows, I would often make icons on my Windows box to run applications on my Fedora box (I think it was FC2 at the time).

A while ago the Docs group was worrying about dealing with the Publican 2/Publican 3 change.They aren't entirely compatible, and you can't install both at the same time.

Oh, and I have a small flock of Raspberry Pis around the shack.
Head smack!

I installed Publican on one of the Pis, and then mounted my document directory from my laptop on the Pi with sshfs.  I then made a one line script that uses ssh to tell the Pi to build the document.

So, instead of
    publican build -l en-US -f pdf
I say


OK, the Pi isn't the fastest thing in the world and it is a bit doggy running Publican, but it gets the job done, and hey, it's running on a whole 'nuther computer so it doesn't interfere with my Mahjongg game.  Life is good.

MPLAB-X -- SDCC -- Fedora

I frequently use MPLAB-X (sadly, not entirely free) for PIC projects.  It works quite well even though it required me to go out and buy a new programmer.  Actually, I got two, both an ICD-3 and a PICkit 3.

Recently I needed to do an 8-bit project, and decided to use SDCC.  It isn't the most efficient compiler in the world, but it is free and I kind of like it.  Since I've recently been doing a lot of 16 and 32 bit stuff, I haven't gotten piklab set up in ages, and besides, it doesn't work with the newer programmers.  And MPLAB-X does have an SDCC plugin.

Unfortunately, MPLAB-X apparently expects an older version of SDCC and it includes a couple of compiler switches that no longer exist.  Although this doesn't cause the build to fail, it does lead to annoying messages which might disguise warnings which I care about.


Although MPLAB-X allows you to add switches, it has no provision for removing switches, and I could not find a way to easily modify the defaults.

I did find that I could edit nbproject/, but as soon as I added any files to the project, MPLAB wrote over the edits.

However, MPLAB provides a Makefile that doesn't get written over, and it has targets to be executed at various times during the build.  By using the .build-pre target, I could edit the other Makefile just prior to the build:


But there is yet another (small) fly in the ointment. Because I have multiple programmers, I often create configurations for each programmer.  Now there is no longer a but instead a and (for the PICkit and ICD).  Fortunately, MPLAB provides a variable, CONF, with the name of the configuration, hence the ${CONF} in the edit.

Now by simply adding a line to the Makefile, my SDCC project builds cleanly and there is no concern about missing important warnings.

Adventures in JNOS/ARM

Sorry, this is long.  More of a tome than a posting.  Originally I hadn't posted this to Planet, but on re-reading it for the thirtieth time I realized it is as much about Fedora as it is about radio.

I thought it worthwhile to document some of my recent trials and tribulations around JNOS.

But for background, it is becoming more and more obvious that digital comms is a key, if not THE key, component of emergency communications.  What is perhaps a bit less obvious is that, while NBEMS modes offer simplicity and much wider acceptance than packet, they are often not up to the task.  Even less obvious is that we need a broader based connection to WinLink.

Many of you may be aware that my personal JNOS is running on a BeagleBoard xM, a small, ARM-based PC, somewhat more powerful than a Raspberry Pi, but available for much longer.  My JNOS is running on Fedora 17.  It works quite well, although is is something of a kludge.
You see, my Beagle's Ethernet port is broken, so it is connected to my LAN via a wireless dongle.  I never got the headless JNOS working on F17, but it didn't matter all that much anyway - WiFi takes a relatively long time to make a connection, so automatically starting JNOS on boot is pretty darned messy.  As a result, I start JNOS manually on the serial port through gtkterm.  The nice part about this is that JNOS doesn't know whether there is actually anything connected to the serial port, so unlike a virtual terminal, it doesn't need to stay connected.

Power to the Beagle is supplied through a little board containing a 7805 and a couple of caps, and a cobbled together coaxial jack connected to the station supply.  The TNC-X is connected via USB.
It took me a while to get the TNC connected over USB; I had long since lost the documentation and there are a number of jumpers to switch.  I was able to write off and get a schematic and eventually get it to work.

OK, that much is ancient history.  The Beagle has worked quite well.  The only real maintenance is restarting JNOS whenever I loose power, and dusting the thing off every few months (I really should get some sort of case).

I do use the mail server on the Beagle/JNOS, and I do all my packet traffic with Evolution (ugly I know, but it is the default).  If you send packet traffic to it appears in my email inbox, just like it should.

Somewhat out of sequence here, for a long time I have been wanting to get soundmodem working.  I recently found the trick (it turns out to be pretty simple), and soundmodem is WAY COOL.  On Linux, soundmodem automatically grabs the AX.25 stack, so all the normal Internet stuff works over packet whenever an address is presented.

My first reaction was soundmodem/Raspberry Pi would make an amazingly cheap JNOS, and since the Pi image is on an SD card, it would be easy to make a JNOS image cheap and easy to configure. Hmmmm....

But then I said, HEY, what about soundmodem on Windows.  It is available for that platform, and might have an even greater impact than a cheap JNOS.  Unfortunately, while soundmodem is available for Windows, Windows doesn't have a AX.25 stack, so it is no more than a cheap TNC. It really doesn't address the complexity, which, IMO, is the major problem.

So, on to JNOS/Raspberry Pi.

An off-topic word about the Raspberry Pi.  This thing truly is a game changer.  You see, when you have a very capable PC that is SO cheap, it becomes realistic to dedicate it to pretty simple applications.  And dedicating a computer to a single application greatly simplifies everything.

I am on my third Pi, and I can see a need for at least two more:

  • Pi #1 is in the Salvation Army vehicle for use in logging during Search and Rescue operations. (MySQL/PHP)

  • Pi #2 is destined to be installed in a local contractor's shop as an electronic sign-out board. (MySQL/PHP)

  • Pi #3 will probably become my JNOS

  • Pi #4 wants to become a media center (MythTV)

  • Pi #5 needed for experimentation on the next new thing

I was able to build JNOS on the Pi, and better yet, I was able to build the headless version and run it as a service.  Very nice. OK, there were some flaky things around running JNOS as a service that I'm not so sure I completely understand, but I can say

   systemctl start jnos.service

   systemctl status jnos.service

   systemctl stop jnos.service

and it all seems to work.  I haven't always been successful with

   systemctl enable jnos.service

but I think that will succumb with a little more beating my head against the wall. (Fedora is just about complete with the systemd conversion, so although the service jnos start stuff still works, it is the old thing, and probably will disappear at some point.)

One hiccup I observed.  In making it headless, I wanted to deconfigure as many options as possible.  I had trouble on F17 making the headless version work, but I had configured in everything I thought I might some day like to play with.  On the Pi I took the alternate approach of configuring out everything I didn't absolutely need.

The problem here is Maiko has a bit of a bug.  At the end of config.h he makes sure that if, for example, you select NETROM off, anything that depends on NETROM is also off.  But he missed something (and of course now I forget what it was!)  Fortunately, this caused JNOS not to build instead of occasionally hanging later, so finding what I missed wasn't too much of an issue.

So, on to running JNOS on the Pi.  Well, first thing I ran into an uh-oh.  The minute I started JNOS the OS hung.  Eventually, I concluded that the system hung as soon as I tried to attach the TNC-X.  I got the same result with the normal Fedora 18 as well as with the new Pidora.

Wondering whether the problem was in JNOS or lower in the stack, I connected the TNC and tried to poke it with gtkterm.  As soon as I selected ttyUSB0 the Pi hung!

The USB chip in the TNC-X is basically a USB to Serial converter, so I tried a couple of others, and no problem with them.  Unfortunately, all the USB to Serial converters I had, all two of them, used the same chip, so I can't be sure, but it looks as if the issue is related to the chip in the TNC-X.

Since this worked on Fedora 17, perhaps I could make it work on some other Rasperry Pi operating system.  I had both Wheezy and Arch Linux for the Pi, so I decided to try them.

Unfortunately, I am sufficiently immersed in Fedora that these other distros are just weird, particularly their package managers.  Wheezy's apt-get wasn't too strange, but I couldn't figure out how to search the repos, and it is clear that the packages have different names than I'm used to, so I couldn't install a package that messes with the serial port, nor could I install enough stuff to build JNOS.  Arch Linux was even worse.  While I could browse their repos on the web and see what was there, Pacman didn't seem to do anything useful.

So, I posted a bug around the USB to serial converter and on to the next idea.

Actually, there are two potential courses here.  JNOS won't use the native AX.25 stack since it has its own, but soundmodem could be a nice, cheap approach. Unfortunately, the Raspberry Pi only has sound out, so it would require a soundcard dongle and some sort of transmit/receive circuitry, so not as simple as I would like.  Still, a transistor and a sound or modem chip might be put on a daughter card pretty cheaply, so that idea isn't totally scratched off the list.

The other approach, somewhat simpler but more expensive, is to use a normal USB to serial converter with a TNC-X.  Along the same lines, the newer TNC-X's use a different USB chip, so they might well work.

I tried converting my TNC-X back to serial, but the result was weird.  When JNOS said to transmit, the transmit light came on but no audio came out.  Evidently, there is some jumper or another that I missed, so the next thing is probably to track that down.

But it looks like I need to order another TNC-X and maybe another Raspberry Pi or two while I'm at it, perhaps even a sound chip.  An even more interesting possibility is to convert the TNC-X code to the dsPIC, which has enough gas to generate and detect the tones itself, making for a single-chip daughter card.  The newer TNC-X code, as far as I know, hasn't been published, but the original has, and I suspect a lot of the stuff in the newer code has to do with his GPS and APRS addons so probably not interesting anyway.

And going back to the top, I really need to find someone to take point on a project to construct a few gateways between our packet network and the NTSD/WinLink network.  Supposedly doable, but not yet demonstrated, and then of course there is the issue of supporting something permanent.

Ahh, so much fun to be had, so little time.

73 de WB8RCR

When it rains it pours

What a wild few weeks.  Lately I haven't been able to spend the time I would like with Fedora, although I did manage to get some release notes work done on some of more geeky beats.  Good thing (really good thing) that others have stood up for the past few releases.  My life shows no signs of getting any more sane.

MiMapOK, the past month or so has been a bit unusual.  We have some nuke drills coming up, well, actually already started, and the State has a new critical incident management system which all the state agencies have been getting familiar with.  We had a "dry run" in the middle of April to familiarize ourselves with the new system, a scheduled meeting of the state agency emergency management coordinators the following day, and the day after that, the State Emergency Operations Center activated in response to flooding across the state.

Although we (amateur radio) didn't have a huge role, still, it took an amazing amount of my time to stay on top of what was happening, participate in SEOC briefings, keep our status updated in the state system, etc.  And the thing about a flood is that it goes on and on and on.  It has now been almost a month and still we are in monitoring mode as the various counties understand the extent of the damage, sort out how they are going to repair it, and who is going to pay for it.

When the government is involved, the who is going to pay for it gets to be a pretty big deal.  There are all sorts of statutes that provide relief for this, that or the other thing, but each has its own set of conditions.  County, State and Federal owned roads are all treated differently, many types of aid are conditioned on having a certain level of damage in a particular city/county/state, etc. etc. etc.  I suppose it shouldn't be a surprise, but it amazes me how many statutes are on the fingertips of State Police lieutenants (the State Police are Michigan's Emergency Management Agency).

Flood4One of the interesting things I noticed was that the damage in the various counties had at least as much to do with the experience and skill of the emergency management coordinator (EMC) for the county as the actual extent of the flooding.  My own county had very extensive flooding, but experienced very little dollar impact compared to counties who saw less extensive flooding.  I see three main reasons for this:

1) Perhaps most significantly, the county experienced a very bad flood about 20 years ago.  Since then construction within the 100 year flood plain was stopped.  As a result, most of the flooding occurred in parks and golf courses.
2) Floods, even flash floods, tend to come with warning.  Our EMC, an experienced and highly skilled guy, was very proactive in preparing, notifying homeowners, getting the right resources in place, etc.
3) We had an exercise around a flood about a year ago, so all the responders knew what to expect.

I find it unfortunate that in Michigan at least, few counties can afford the kind of emergency management skill they need.  That is aggrivated by a culture in much of Michigan that is decidedly anti-government.  There are a number of counties where half the official positions are held by the same person, and the other half by his brother in law, simply because nobody else is willing to take the job.

Of course, the flood isn't the only thing consuming my time lately.  I had a laptop issue that caused me to re-install Fedora, and of course, there are dozens of things not yet whole.  Add to that my main server is in similar shape, and it seems like every time I turn around there is some new bit of software that no longer works, or something I forgot to install.

Plus, in the midst of all this we had a meeting of all the amateur radio Emergency Coordinators.  Since I was the instigator, almost all the preparation was on me.  And the same thing with an exercise we had last weekend.  And of course the release notes coming due.  Yeah, mostly I do it to myself, but a lot of things were on the books for April/May and getting consumed by a flood was the last thing I needed.

And I'm not sure why, but a lot of things are suddenly falling out that in more sane times would have been welcome.  Just before all this started I met with the Public Service Commission to talk about communications in the event of a widespread critical infrastructure outage. The State Police are wanting to get their database of communications assets updated with amateur radio assets,  we are about to put our training data into the State's database, the State is asking if we can perhaps provide video from their aircraft, and I was tasked with lining up some training for a FEMA official in Washington.  On top of that the Emergency Communications Advisory Committee of the ARRL is now meeting frequently to get some of our work wrapped up, and I've just been appointed to a committee that develops public safety communications policy for the state.

I guess it is true when they say there is no rest for the wicked.
  • Current Mood
    satisfied satisfied

A teeny web server for Search and Rescue

For a while now I have been working with Midland County Search and Rescue, operating as net control on the radio net we use to keep track of the teams in the field.  This turns out to not be quite normal amateur radio logging; the key issue is accountability - making sure we know where everyone is and that they get back safely.

SAR-loggingAfter doing this for a while, I realized the job could be made easier with the help of a bit of PHP, so I built an application that helps me keep track of the teams, and also prepares a PDF of the log after the search.  Often on a search there are law enforcement issues, and it is important to have proper records.

I generally run Apache/MySQL on my laptop, so this works quite well, and easily.  But when someone else is net control I either have to let them use my laptop, or have it become a server.  The group has a number of laptops, but they all run Windows, and I wasn't interested in dealing with the complexity of SQL Server and all the weird Microsoft server page things.

Raspberry Pi in Bud boxBut, the Raspberry Pi provides a really beautiful answer.  I installed the necessary software on my Pi and it is surprisingly snappy, even running MySQL under the web server.  Since it is a simple matter to make the database and web server come up on boot, I realized that this thing could be made incredibly simple.  So, I put the Pi in a Bud box with holes for the USB, ethernet and power.  The USB sticks out so to make the RJ45 reachable I had to expose it, requiring the third hole.

USB connector covered with tapeSetting up for a search is kind of a messy business, with all sorts of things going on, and most of the folks are not at all technical.  So I wanted to be sure it was as simple as possible.  With the Pi in a box and very little exposed, all that is required is to plug it into power and a router and it's done.  I had to expose the USB connector, which is unnecessary for this application, so I taped over the USB port, leaving no room for error.  The RJ45 cable won't fit into the hole for the power, and the little micro-USB will just rattle around inside the RJ45 jack, so there isn't any opportunity for someone to make a mistake.

This also allows some additional features.  Since we will now always put it on our local, scene WiFi, search operations will be able to view the log live.  I plan in the future to come up with another page which shows an expanded team status.  Currently I have a little snapshot of the team status on the upper right of the logging screen, but I have additional data which would be interesting to SAROPS so I plan to add that as well.

The Raspberry Pi, of course, is running Beefy Miracle. 

Wah hoppen to jjmcd?

OK, this is long overdue.  As many of you are aware, I am not leading the charge on the Release Notes this release.  And although I've been hanging around on IRC quite a bit, I rarely have much to say these days.

It ain't that I don't still love ya.

This fall a number of things have colluded to make it impossible for me to spend as much time as I would like on Docs.  I do anticipate things will get a lot better after October.

Many of you know that I am the Section Emergency Coordinator for Michigan.  That has me spending a lot of time in Lansing with the State Police.  This is nothing really new, *BUT* ...

This year the State of Michigan is rolling out a new Critical Incident Management System.  I have been quite involved in the development of that system, seizing the opportunity to ensure that features are included which are useful to amateur radio.  This, of course, takes time, and a lot of that time is surrounded by two hour drives each way.

Perhaps instigated by this rollout, my responsibilities with the Michigan State Police have increased.  Instead of simply coordinating amateur radio activities, I now have responsibility for all auxiliary communications for emergency response.

Also on the government front, every eight years each nuclear power plant is required to run what is called an "ingestion pathway" exercise.  Since Fukishima, the NRC has really ratcheted up these exercises.  This is the first that Michigan will be doing since that event.  When we do any of these nuclear plant exercises, we do two drills and then the actual exercise.  During the drills there is less going on out in the field, but from my perspective in the State Emergency Operations Center, they are all the same.

Except in this case, the final exercise will be multiple days instead of one. In the past, even when we had an ingestion pathway exercise, I wasn't involved in the second day anyway.  This time I'm not so sure.  In fact, we have little insight into what to expect except a LOT more Federal participation than we are used to.  I was at a "Federal Outreach" a couple of months ago, and it sounded ominous.

The first drill is tomorrow.

Saturday I am speaking at the Ohio Linux Fest.  Of course, getting ready for that takes time, and me being me, probably takes more time than it needs to.  The good news is that next week that will be behind me.

Speaking of OLF, where are the Fedorans?  I don't see any familiar names on the speaker list where in previous years it looked like a FUDcon. I don't think I even know of anyone else who is going.

But the big time eater is SET.  Every year amateurs across the nation hold what is called a Simulated Emergency Test.  Typically, this is a pretty low impact event, but in Michigan, we have chosen to use this as an exercise to help us improve our skills.

Over the past couple of years, we have begun using FEMA's Homeland Security Exercise Evaluation Program (HSEEP) for our exercises.  As you might imagine, this adds a huge amount of work to preparing for and evaluating an exercise.  However, the results are more than worth the effort.

Most times, I have an assistant who does much of the heavy lifting.  However, this year he has a number of personal demands on his time, so he hasn't been able to assist much.  The task takes a fair bit of training, including a number of FEMA courses, so is isn't something I can quickly grow someone into.

Once I get past October, things should look a little more normal.  But I don't know, I need to keep doing it to myself.

Back in 2003 I did an online course on PIC microcontrollers.  The course was amazingly well received; thousands of folks took it.  When we were first talking about it, I was thinking we might get a dozen.  One of the QRP clubs kitted a board to go with the course, and after a few months they cried uncle; they just couldn't keep up.  Eventually the board was commercialized, and it is still available and still selling.

PICs have changed a lot, and I am toying with doing a course on the newer dsPICs.  These have way more capability, are a lot easier to use, and cost barely more than the old ones.  With the help of a couple of other folks, I've been working through what a board would look like if I were to do this.  And then there's the whole deal of what will it cost to manufacture, is there enough of a market, etc. etc. etc.

The good news is that after October, the SCHEDULED items die down, and hopefully things will return to their normal level of chaos.

Meanwhile, many thanks to all the folks who have stepped up to be sure the Release Notes are the best ever.