Hey all,
The server has, as you've surely noticed, been down for a couple of hours. I'm still investigating why. The only think I know right now is that it just stopped responding to everything but purely kernel-based communication (like pings and SYNs), so I'm suspecting it came under heavy load for some reason. It may be related to an enormous spike of incoming traffic that went on for some hours before it went down (so it might be a DoS, though I'm not drawing that conclusion yet).
I have hard-reset the server using Hetzner's web interface. I wanted to avoid doing this, since it risks causing the same kind of data corruption as some of you probably still remember from when Jorb's server went down due to power outages before we got a UPS from Granger for it.
I'll be investigating the causes and effects now, and update this thread appropriately.
Update 1: No map files seem to be truncated, so it'll probably "just" be a normal rollback, at least. (Subupdate: Not so; see updates 3 and 4 below)
Update 2: The failure appears to have been caused by Apache eating metric shittons of memory. This may be caused by me forgetting to set up the minimap directory to forbid index listing; someone might have requested it, even by mistake.
Update 3: Many village files have been corrupted. :(
Update 4: The corrupted villages will appear as if gone, quite simply. Their village idols can be reactivated as if they were newly built. I'll try to arrange so that broken authority objects can be right-clicked to turn back into the materials from which they were built.
I hope that's all for the updates, now.