After the power cut.

Power was out from 9am (which was as expected) until 11:30 (which was a lot sooner than I was expecting). Just before the power cut, I went round powering down computers that I felt I could live without, and I strung the power wires from the generator to where power was needed, including up in the attic!

As soon as power was cut, I started up the generator (what a noise!) and plugged things in. Several problems soon became apparent.

The biggest problem was that although some of the UPSes were happy to take power from the generator, others weren't. As a rough rule of thumb, the small cheap UPSes were fine, but the big professional ones weren't. I tried putting an isolating transformer between the generator and the UPS, but that didn't help (I hadn't really though it would, but it was easy to try).

So I googled this problem. I think the issue is that the generator (being a cheap petrol 3000 watt thing) makes very dirty power. The power that comes out of your wall, is a nice clean sine wave, smooth and neat, 240 volts plus or minus a couple, and 50 hertz. But what comes out of the generator isn't 240 volts, it's all over the place, the frequency isn't 50 hertz, it's whatever the generator feels like and far from being a nice smooth sine wave, it's jagged and horrible. And the UPSes didn't like that, and kept switching to battery power, staying there for a minute, tasting the generator power and deciding they didn't like it and going back to the battery. Which meant that the batteries were slowly running down.

The solution is this. On the back of an APC Smart-UPS, there's a button marked "sensitivity" and a green LED. By default, they're on high sensitivity, and the LED is bright. Press the button once, and you get medium sensitivity, and a dim LED. Press it again, and you get low sensitivity and the LED isn't lit. And press it a fourth time and you're back to high sensitivity.

I did some experimenting, and medium sensitivity is probably good enough; you can drop down to low if it isn't. I found that the APC UPSes were happy with the generator at medium sensitivity.

The other UPSes that I have, don't seem to have this facility. Well, that means I'll be leaning more towards APC in future.

I've got two APCs in action, and I've got four more dead ones. I'm pretty sure they're only dead because the batteries are duff, so I've ordered a set of batteries (£95, but a whole new UPS would be three or four times as much) and I'll try to revive at least one of them.

The Raspberry Pies were heroes during the outage - they use so little power, I didn't take any of them off the UPSes. That was nice - it meant that four mail servers, two DNS servers and various other stuff carried on regardless. But I did take down my internal nameserver (oops), which meant that I had to remember IP addresses to access things. I've now configured one of the Pies to act as an internal nameserver using bind9. Also, I took down my DHCP server, which meant that computers couldn't join the network with dynamic addresses, and a few of my computers do exactly that. So I've also set a Pi up to be an extra DHCP server, using DNSMasq.

The UPS protecting my main router and firewall was fine, and was happy with the generator power.

When power came back, one of my servers wouldn't start - the problem turned out to be the power supply in the server; a quick and easy fix. Another server made a dreadful noise when it was starting. I tracked that down to a noisy fan, which I've replaced. A couple of others just wanted to be rebooted again before behaving themselves.

So I survived the power outage with no big problems, and I've learned a few things as a result.