I've got a big file. It's 26 gigabytes; it's part of a database system I wrote. To write it, I invented 40-bit integers - you don't want to know about it. Even the indices are indexed.
Anyway - I wanted to copy this big file, becasue the drive it's currently living on is reporting deterioration to an extent that I think it might fail within a year. I've got backups, of course. But I don't want to wait until it actually fails, because it'll fail at the most inconvenient time possible, because computers do that.
So I installed a 2tb drive, formatted it, and tried to copy the file. I tried with cp. I tried with scp. I tried with lftp. In all cases, the copy bombed out at 17 gigabytes, and I couldn't think what the problem might be.
Then I realised - it isn't 17 gb, it's 16 gb. It looks like 17 because I'm looking at it in decimal. And then I realised, I'd formatted that 2tb drive with 1kb blocks, because I was also expecting to store a lot of small files. But on an ext3 file system, if you have 1kb blocks, them maximum file size is 16gb. Bingo. So I'm formatting it to 4kb blocks, and then my maximum file size will be 2048 gb and I'll be a happy bunny.
Oh, and I also discovered something else. When I tried to scp that big file from another computer, it was copying at a rate of 9mb/second, which sounds great, except that I'm running a 100 mbit network, and 9mb/sec is 90 mbit. So that file copy was saturating my network, and nothing else could happen.
So when I tried to log into another server, nothing happened. And this was a server I've just upgraded in memory, so I thought it had crashed with a memory problem. In the machine room, I connected a monitor to that server, and the screen was blank. And I hit the keyboard a couple of times in case it was the screen-blanker, but it stayed blank.
So then I spent an hour swapping out memory, putting heat spreaders on memory, and putting the server back on the shelf.
Later (the next day) I noticed that the keyboard cable had come unplugged, so my hitting the keyboard wasn't deactivating the screen blanker. The computer had been working fine all along, it was just my self-imposed network saturation that made me worry, and the break in the keyboard cable confirmed my worries. Wrongly.
So in future, to avoid that problem, when I scp a huge file that's going to saturate my network for a long time, I'll use the -l option, as in -l 50000 so that the copy uses less than half my network capacity.