Pages

Sunday, 24 July 2016

Server becomes readonly

I have a server where the main drive has recently started to become read-only, a few times per week. What causes that? There's probably several possible causes.

I looked at /var/log/messages, which was a huge file, 92 megabytes. It's the system log. So I didn't read it line-by-line, I used grep, which roughly translates to "look for something".

I did:

grep "filesystem read-only" /var/log/messages

Actually, before I did that, I did readonly, and then read-only. Anyway, it gave me this:

Jul 19 16:53:46 usula kernel: EXT4-fs (md0): Remounting filesystem read-only
Jul 19 18:04:45 usula kernel: EXT4-fs (md0): Remounting filesystem read-only
Jul 22 07:18:56 usula kernel: EXT4-fs (md0): Remounting filesystem read-only
Jul 24 00:11:34 usula kernel: EXT4-fs (md0): Remounting filesystem read-only


That told me exactly when it happened, so then I looked at the segment of the log at around that time, and it told me that the problem was a disk error on drive sdb. Linux couldn't write to the journal, so it remounted the drive as readonly. Which is a reasonable reaction, but it does leave me with a problem.

So here's what I've done.

1) I've brought up another server, which is a copy of usula, and I've set up a cron job on it so that it's updated every two hours. That's in case I decided to retire usula.

2) I've set up a cron job on usula that greps the system log once per minute, and emails me if it finds "filesystem read-only", so at least I'll know very soon after it happens.

No comments:

Post a comment