Most of my Pies just keep on running. They'd been going for about two years nonstop before they suddenly all died. The fault was the power supply that fed them; the fan had died (fans are a weak point in any system - being mechanical, they wear out), the PSU had overheated and shut itself down.
But the two Pies that I use to tell me the outside temperature, give me a view outside, the time and the number of alerts on my server monitoring system - crash much more often. Not a big deal, it only affects me (and only slightly) and all I have to do is cycle the power so they're up and running again.
Wouldn't it be nice to automate this? I already have for the more important Pies, that carry email and DNS.
I did it slightly differently this time. I started with four female USB ports, and I soldered then to a strip of metal, so they were nice and stable. The outside connectors are the positive and negative, and that was all I was going to use. I soldered leads to the negative connectors, and put all the four leads into one terminal connector. On the positive side, I soldered a lead to each connector, and the other end of that went to a serial relay box. I've used these before for all sorts of purposes (the robot arm uses one, for example), and when I bought them, they were less than a third of the price they're shown at now. You can get something similar on Ebay for £10. A similar thing to run off USB ports is £19.
A long long time ago, to start up a PC you had to apply power, and then press a button at the front. These days, you can set u a PC so that no button press is needed. But I wanted to be able to power-cycle my PCs remotely before that was possible. I wired up that button-press so that it could be done by these relay boxes. Pretty cool! So I have a few of these left over from those days, and I would never throw away something as potentially useful as that.
I connected the four positive leads to the "common" connector of each relay, and I connected the four "normally closed" relay connectors to another terminal connector. The effect of this is that if I feed that terminal connector with five volts, then five volts will appear on all four USB connectors. But if I switch one of the relays, then the power for that connector is switched.
I got the five volts from a step-down (buck) converter, that costs £1 on Ebay, but I prefer to use the ones that display the voltage, costing £2.
The feed into the converter is the 12 volt line from an old PC power supply, and the same 12 volts is also used to power the relay box. So, using this, I can run up to four Raspberry Pies off the same power supply, and have remote power-on, power-off control.
So next, to automate this. The relay box is connected to the serial port of a computer, and on that computer, I run a program once per minute. that program pings the Pies, and if it doesn't get a good reply back, it then cycles the power to the Pi that isn't responding.
To cycle the power, I use a program called Minicom, which is great for this purpose. It's a terminal emulator. I set it to 9600 baud, 8N1 which is what the relay box uses, and I set up a file called "off01" which will control relay number 1. So, to switch it off, that's
! killall -9 minicom
And to get minicom to do the job, my perl program does this:
`minicom -o -t ansi -S /home/drsolly/engine/off01`;
"N1" means "switch relay 1 on" but because I have them wired as "normally on", it means "switch relay 1 off" And "F1" means switch it on. So when it can't contact that Pi, it powers off, waits five seconds, then powers on. Of course, if the Pi doesn't start up within 60 seconds, the next time the program runs it will do the same thing again ... and again ... and again ... but I know from experience that it takes less than 60 seconds to start up a Pi.
I have it running for one of the Pies; after that's been going for a while, I'll connect the other Pi to the same box, and I still have two spare slots.