Downtime notice: Server overload

[loftar]

Hey all,

The server has, as you've surely noticed, been down for a couple of hours. I'm still investigating why. The only think I know right now is that it just stopped responding to everything but purely kernel-based communication (like pings and SYNs), so I'm suspecting it came under heavy load for some reason. It may be related to an enormous spike of incoming traffic that went on for some hours before it went down (so it might be a DoS, though I'm not drawing that conclusion yet).

I have hard-reset the server using Hetzner's web interface. I wanted to avoid doing this, since it risks causing the same kind of data corruption as some of you probably still remember from when Jorb's server went down due to power outages before we got a UPS from Granger for it.

I'll be investigating the causes and effects now, and update this thread appropriately.

Update 1: No map files seem to be truncated, so it'll probably "just" be a normal rollback, at least. (Subupdate: Not so; see updates 3 and 4 below)

Update 2: The failure appears to have been caused by Apache eating metric shittons of memory. This may be caused by me forgetting to set up the minimap directory to forbid index listing; someone might have requested it, even by mistake.

Update 3: Many village files have been corrupted. :(

Update 4: The corrupted villages will appear as if gone, quite simply. Their village idols can be reactivated as if they were newly built. I'll try to arrange so that broken authority objects can be right-clicked to turn back into the materials from which they were built.

I hope that's all for the updates, now.

[Darki]

*Gets his hopes up to time travel to world 2*

[Tacheron]

Would it be possible to know, once you've resolved the issue, how much of a rollback we've had? If it's too big of an issue, then forget about it, but if it's not, I'd like to have that info

[loftar]

Rollbacks are, by their nature, not homogenous. Map parts are saved as they are unloaded, but those that are kept loaded by the presence of people are only saved once about every two hours. It shouldn't be any longer than that, in any event.

[fdse2]

So... maby add some new stuff to make us happy again:3

[jongre]

Ah, I wont loose that much

[belondes]

alright everyone from my village was kicked from it somehow

[Fratari]

The actually village of lost caverns is simply gone. I'll edit when i manage to check where our idol should be.

Edit: There goes the server.

[belondes]

and we got a rollback for about ~30 mins

[zacty]

and we got a rollback for about ~30 mins

cool, all i lost was me carrying a loom to the other side of my house