Granger wrote:Then put some of the donations into a SSD as main disk (to get throughput and IOPS to keep game server served), use LVM snapshots (or ZFS clones or whatever) to freeze a filesystem copy and then rsync it to a normal RAIDed volume for backup (using hardlinks against an old backup will offer multiple generations cheaply).
Well, true enough, there are ways around it. I haven't exerted any efforted for it, however, since I've mainly just kept a backup around for the utmost disaster of a hard drive failure, though. I hadn't expected XFS to regularly corrupt data on power failures. Reasonably, I guess I should switch back to ext3 instead (I used XFS on the new hard drive mainly for its ability to be extended on-line, and because I find it performs rather well).
Unclean shutdown of a filesystem is never a good idea.
Maybe check the mount flags, might be that you disabled something by accident which leads to unrecoverable state when shit happens.
As far as i have seen ext3 with journal is most reliable in case unclean shutdown (UPS with OK battery pack will be send on its way tomorrow, UPS - the mail guys - website tells me it should arrive on friday). Putting a call to sync into cron also helps a bit (but avoiding unclean shutdown should have priority over any workarounds).
My suggestion still is to put some € into an SSD for game data (to get low latency) and cron a niced rsync (with --link-dest option against the last backup, just put something into the script which dumps oldest backup(s) in case space on backup drive starts to run low), maybe even from a temporary snapshot to avoid dupe bugs, to a time where server load is low.
In case you feel that server performance would still suffer from this then put a sign on the website saying every day at x:00 UTC background backup will run so performance will suck - but since you won't suddenly miss your village because of existing current backups (in case the server crashes hard for some reason) you will want to live with that without complaining
EDIT: shit, wrong account. Greetings from Granger.