Server hardware problem

[loftar]

The server is experiencing some hardware problem, the more exact cause of which is still undetermined. One of the hard drives (the one with the actual game and account data on it) has stopped responding. I will contact the host (Hetzner) and see if they have any clue what's happening, but until then, all related services are by necessity down, including not only the game, but also the account server, which means that you won't be able to log in on the site (and therefore not the forum either). You will be able to stay logged in as long as your cookies last.

It is quite unclear what is causing the hard drive to fail. It looks a bit like a SATA communication error (rather than complete hardware failure), but not having local access, I cannot really determine the cause. Interestingly, the other hard drive is currently 58°C. It might be some local problem at Hetzner.

In the worst case, the hard drive is fried and everything on it is lost. In that case, that would include not only the current world, but all accounts as well. I would restore the ones that were on Jorb's server when Haven left it, but that would still leave a lot of destroyed accounts. I sincerely hope it won't have to go that far.

Update, 13:45 UTC:
I powercycled the server properly, and it did detect the hard drive again, and the accounts look safe, at least. There is still a problem in that the XFS logging behavior that I have described elsewhere has nuked some parts of the map, however; and because I suck, I have no backup of them to restore. I'll be looking to see if there some way to recover such files from XFS. Until then, login will remain disabled (since the filesystem is unmounted).

After that, I'll be looking into doing backups. :P

Update, 17:30 UTC:
It does not seem as though I will be able to retrieve the nuked grids. See this post for a list of affected areas.
EDIT: Also, hearth secrets are reset and charter stones will need to be reactivated.

Update, 23:20 UTC:
The server has been down a few hours for filesystem reconfiguration. I've moved the root filesystem over to LVM/RAID1. I will move the game data filesystem only now that the forums are up again, and it's probably going to take a little time, but I don't know exactly how much yet. After that, it should be good to go again. The the accounts and maps in their current state have also been backed up to Jorb's server.

Update, 01:20 UTC:
The accounts have been copied from the old filesystem to the new now, so login to the website should possible again. I will now copy the maps.

Update, 02:00 UTC:
The server is running again, for now. Let's see how it goes. EDIT: A couple more nuke-grids detected; nuke-post updated accordingly.

[Kaios]

go to fuk

[Saphireking65]

so I shouldn't close my browser window until all this is settled?

Also, only Alleyn, Assassin777, bmjclark, boshaw, Dataslycer, DatOneGuy, Evarine, Frelghra, Grimreepr, Haba, Jotun, loftar, ninja_yodeler, novaalpha, sergo, snami, virus_128, Windforce, and I can post right now :X

[novaalpha]

w6 get

[bmjclark]

so I shouldn't close my browser window until all this is settled?

Also, only Alleyn, Assassin777, bmjclark, boshaw, Dataslycer, DatOneGuy, Evarine, Frelghra, Grimreepr, Haba, Jotun, loftar, ninja_yodeler, novaalpha, sergo, snami, virus_128, Windforce, and I can post right now :X

I feel kinda special ^.^. Im also never closing my browser

[burgingham]

so I shouldn't close my browser window until all this is settled?

Also, only Alleyn, Assassin777, bmjclark, boshaw, Dataslycer, DatOneGuy, Evarine, Frelghra, Grimreepr, Haba, Jotun, loftar, ninja_yodeler, novaalpha, sergo, snami, virus_128, Windforce, and I can post right now :X

Don't forget the hidden ones.

[ninja_yodeler]

oh no's

could you please post the cut off date (if i understood correctly) to which the accounts could be lost. as in before ?/?/??


also inb4 RUA did it

EDIT: every post before this ninja'd me. slow typeing by me much

[Saphireking65]

so I shouldn't close my browser window until all this is settled?

Also, only Alleyn, Assassin777, bmjclark, boshaw, Dataslycer, DatOneGuy, Evarine, Frelghra, Grimreepr, Haba, Jotun, loftar, ninja_yodeler, novaalpha, sergo, snami, virus_128, Windforce, and I can post right now :X

Don't forget the hidden ones.

Oh you.

[Patchouli_Knowledge]

Do we have an estimated time of notice of knowing what happens to the HDD?

[DatOneGuy]

Very interesting stuff here.