I have a server where the main drive has recently started to become read-only, a few times per week. What causes that? There's probably several possible causes.
I looked at /var/log/messages, which was a huge file, 92 megabytes. It's the system log. So I didn't read it line-by-line, I used grep, which roughly translates to "look for something".
grep "filesystem read-only" /var/log/messages
Actually, before I did that, I did readonly, and then read-only. Anyway, it gave me this:
Jul 19 16:53:46 usula kernel: EXT4-fs (md0): Remounting filesystem read-only
Jul 19 18:04:45 usula kernel: EXT4-fs (md0): Remounting filesystem read-only
Jul 22 07:18:56 usula kernel: EXT4-fs (md0): Remounting filesystem read-only
Jul 24 00:11:34 usula kernel: EXT4-fs (md0): Remounting filesystem read-only
That told me exactly when it happened, so then I looked at the segment of the log at around that time, and it told me that the problem was a disk error on drive sdb. Linux couldn't write to the journal, so it remounted the drive as readonly. Which is a reasonable reaction, but it does leave me with a problem.
So here's what I've done.
1) I've brought up another server, which is a copy of usula, and I've set up a cron job on it so that it's updated every two hours. That's in case I decided to retire usula.
2) I've set up a cron job on usula that greps the system log once per minute, and emails me if it finds "filesystem read-only", so at least I'll know very soon after it happens.