For no apparent reason, Facebook has restored my account
I got my first warning, when part of an important file, that was text-only, became corrupt and had binary stuff in it. I was able to replace the binary from a backup.
Then it happened again. And again. I'm so glad that I use a flat-file text-only database.
This is an important computer, it's the one I do billings from, and is therefore inside the innermost bastion of my firewall. That turned out to be a big problem later.
I used the SMART system, which tells me how many bad sectors there are on the drive, and it was large - and growing fast. This disk was on the verge of failing.
So, first I built a replacement computer. I used an Intel motherboard (I bought a job lot of these a few years ago, and they're great) with the CPU an E7500 dual processor running at 2.93 GHz. I put in 8gb of memory, and an 80gb 2.5 inch drive.
Next, to configure it, I used a 192.168.2.2 ip address, because that's in the range I use for "inside", the innermost bastion. That meant that I couldn't connect it to the usual 10.x.x.x range, it had to connect to the "inside". But the switch that is fed by the "inside" is way across the room, so I cleverly put a switch between the firewall "inside" and the switch across the room. Everything still worked. So I took a feed from the new switch.
That should work, right?
But it didn't. And I still don't know why.
I tried lots of things, such as replacing the new switch, and many others that didn't work.
Eventually, I did a "hail Mary" (an American football term for a desperation move) and slung a 10m cable from the old switch right across the room to the new server, and to my utter astonishment, it worked!
So I loaded up the new server, partly from the failing old one and partly from the backup, installed apache (latest version, and configuring that led to much grief).
Eventually, I got everything working, put the new server in the place where the failing server was, and everything is OK now.
First, I'll explain my email system. It's complicated.
To tell other computers where to send email for me, we use the DNS (Domain Name Service) system. In that, an MX (Mail eXchange) record tells other computers where to send my email. There are two main servers, mail2 and mail1. The MX record tell the priority. So, mail is sent to mail2, unless mail2 isn't working, in which case it goes to mail1. Both are Raspberry Pis, version 1, vintage 2012. I also retreive email from several AOL and Gmail accounts.
Mail1 gets spam, nearly everything. Because it shouldn't be used unless mail2 isn't working. I take advantage of this by setting up a bunch of MX records with even lower priority as spam traps.
But I don't want to visit both computers, so I set up fetchmail, to visit each in turn and use the IMAP protocol to collect unread mail and put it on my email reading computer., xantl. To read mail I use pine or alpine, a text-only reader. That means that I don't have to worry about unpleasant surprises that you can get in web-based readers.
When it gets to xantl, it passes through my mail spam filter,sorting the mail.
So what went wrong? Suddenly, fetchmail stopped fetching mail from mail2 and mail1. I hadn't made any recent changes. This happened a few weeks ago.
So I tried fetchmail -vvv (verbose) but that told me nothing useful.
I tried reinstalling fetchmail. i tried installing fetchmail on another computer. I tried reinstalling dovecot (the IMAP server) on the pis. nothing worked. I tried updating OpenSSL - no joy.
I tried using .forward on mail1, but that didn't help. and all this while, I was reading an dealing with emal on mail2 and mail1.
I tried changing the MX records to a different computer. No use.
Then I did what I should have done in the first place. Google is a great resource, but I also have a file where I keep a record of past problems and thier solitions, and I found this gem.
To see what fetchmail is doing:
fetchmail -Nvvvd0 --nosyslog
So I did that, and it told that the problem was that fetchmail couldn't negotiate a secure channel with mail1. But I don't need a secure channel because the whole network is secured.
So I added
to the .fetchmailrc, telling fetchmail not to use secure sockets.
And everything worked!
I think this happened because fetchmail updated itself without telling me. The fact that a couple of weeks of Googling turned up nothing, makes me think that I'm the only person who has ever had this problem. There can't be many people using fetchmail release 6.4.23 to retreive mail from 10 year old Raspberry Pis.