Downtime notice: Server overload

Announcements about major changes in Haven & Hearth.

Downtime notice: Server overload

Postby loftar » Sun Nov 28, 2010 3:56 pm

Hey all,

The server has, as you've surely noticed, been down for a couple of hours. I'm still investigating why. The only think I know right now is that it just stopped responding to everything but purely kernel-based communication (like pings and SYNs), so I'm suspecting it came under heavy load for some reason. It may be related to an enormous spike of incoming traffic that went on for some hours before it went down (so it might be a DoS, though I'm not drawing that conclusion yet).

I have hard-reset the server using Hetzner's web interface. I wanted to avoid doing this, since it risks causing the same kind of data corruption as some of you probably still remember from when Jorb's server went down due to power outages before we got a UPS from Granger for it.

I'll be investigating the causes and effects now, and update this thread appropriately.

Update 1: No map files seem to be truncated, so it'll probably "just" be a normal rollback, at least. (Subupdate: Not so; see updates 3 and 4 below)

Update 2: The failure appears to have been caused by Apache eating metric shittons of memory. This may be caused by me forgetting to set up the minimap directory to forbid index listing; someone might have requested it, even by mistake.

Update 3: Many village files have been corrupted. :(

Update 4: The corrupted villages will appear as if gone, quite simply. Their village idols can be reactivated as if they were newly built. I'll try to arrange so that broken authority objects can be right-clicked to turn back into the materials from which they were built.

I hope that's all for the updates, now.
"Object-oriented design is the roman numerals of computing." -- Rob Pike
User avatar
loftar
 
Posts: 8926
Joined: Fri Apr 03, 2009 7:05 am

Re: Downtime notice: Cause yet unknown

Postby Darki » Sun Nov 28, 2010 4:04 pm

*Gets his hopes up to time travel to world 2* :lol:
User avatar
Darki
 
Posts: 363
Joined: Mon Aug 24, 2009 11:19 am
Location: Cow farm :D

Re: Downtime notice: Cause yet unknown

Postby Tacheron » Sun Nov 28, 2010 4:05 pm

Would it be possible to know, once you've resolved the issue, how much of a rollback we've had? If it's too big of an issue, then forget about it, but if it's not, I'd like to have that info ;)
This is not a test of power
This is not a game to be lost or won
Let justice be done
User avatar
Tacheron
 
Posts: 837
Joined: Fri Apr 16, 2010 12:11 pm

Re: Downtime notice: Cause yet unknown

Postby loftar » Sun Nov 28, 2010 4:08 pm

Rollbacks are, by their nature, not homogenous. Map parts are saved as they are unloaded, but those that are kept loaded by the presence of people are only saved once about every two hours. It shouldn't be any longer than that, in any event.
"Object-oriented design is the roman numerals of computing." -- Rob Pike
User avatar
loftar
 
Posts: 8926
Joined: Fri Apr 03, 2009 7:05 am

Re: Downtime notice: Server overload

Postby fdse2 » Sun Nov 28, 2010 4:17 pm

So... maby add some new stuff to make us happy again:3
fdse2
 
Posts: 25
Joined: Sun May 30, 2010 11:10 am

Re: Downtime notice: Server overload

Postby jongre » Sun Nov 28, 2010 4:18 pm

Ah, I wont loose that much :)
User avatar
jongre
 
Posts: 94
Joined: Sun Nov 07, 2010 12:18 am

Re: Downtime notice: Server overload

Postby belondes » Sun Nov 28, 2010 4:26 pm

alright :) everyone from my village was kicked from it somehow
User avatar
belondes
 
Posts: 25
Joined: Sun Oct 24, 2010 6:03 pm

Re: Downtime notice: Server overload

Postby Fratari » Sun Nov 28, 2010 4:28 pm

The actually village of lost caverns is simply gone. I'll edit when i manage to check where our idol should be.

Edit: There goes the server.
I think it's made of bubbles and hate.

Sarchi wrote:Only time we'll track scents is if it appears that you intentionally tried to vandalize our stuff, then we'll raid your town and kill you and everyone that lives in there because fuck you. We'll probably even take your cupboards afterwards, lol.
User avatar
Fratari
 
Posts: 223
Joined: Fri Feb 12, 2010 3:01 am

Re: Downtime notice: Server overload

Postby belondes » Sun Nov 28, 2010 4:32 pm

and we got a rollback for about ~30 mins
User avatar
belondes
 
Posts: 25
Joined: Sun Oct 24, 2010 6:03 pm

Re: Downtime notice: Server overload

Postby zacty » Sun Nov 28, 2010 4:34 pm

belondes wrote:and we got a rollback for about ~30 mins


cool, all i lost was me carrying a loom to the other side of my house :D
i really need to find a good signature
User avatar
zacty
 
Posts: 1023
Joined: Fri Nov 26, 2010 2:58 pm
Location: somewhere over the rainbow

Next

Return to Announcements

Who is online

Users browsing this forum: No registered users and 6 guests