Server Issues: Crashes, Lag, Restarts, Rollbacks

Announcements about major changes in Haven & Hearth.

Server Issues: Crashes, Lag, Restarts, Rollbacks

Postby jorb » Sun Jan 24, 2010 9:16 pm

To avoid the speculation that's been going on over the state of our servers, here's the real story, all collected in one.

Any future discussion on server status will take place in *this* thread. From here on to eternity, or the next official post on the subject. Threads will not be retroactively merged, because there's a lot of BS in the other threads. I will keep the latest official server status as a sticky at all times. Keep it out of the Inn and Bugs.

1) Server is, performance wise, tagging along just fine. Hard drive, RAM and CPU are handling the recent influx of players very well with some trivial optimization of map load/save times etc.
Loftar edit: When Jorb says "tagging along just fine", he really means that it isn't currently breaking. However, it should not be construed to mean that there isn't the slightest bit of strain on it. It wouldn't manage any vast amount more players right now. If I were to guess, things might start getting a bit ugly around 300-400 simultaneous players, but maybe earlier depending on how various algorithms turn out to scale. I do have plans for optimizing various parts of the server code, mind you, but I haven't done so yet, and doing so will take a bit more time than I have right now.

2) We have had some massive bandwidth problems lately. Technically we should have 10 Mbps sending band -- we're currently sending *2.5 -- but for some reason (Which loftar is working on) we can't seem to get much more output than that. While this remains an issue, we will have the lag. We are working on this. This is a priority.
Loftar edit: As noted later on in this thread, I have a potential fix for this now, but it hasn't been verified to work yet due to too few players being logged in right now. We'll see if it works once the russians get back online.

3) We do not believe we have been DoS:ed. We're certainly not under attack. We've had two massive bandwidth-spikes, but they do not seem to be related

4) Heirs of Rurik are, as a matter of simple course, very welcome in the H&H game, and forums. We are proud to have you, please bring your friends.

Loftar can fill this in with actual tech if he wants to. I suggest he do that as marked comments entered in as edits of this entry. Over and out.
"The psychological trials of dwellers in the last times will be equal to the physical trials of the martyrs. In order to face these trials we must be living in a different world."

-- Hieromonk Seraphim Rose
User avatar
jorb
 
Posts: 18263
Joined: Fri Apr 03, 2009 7:07 am
Location: Here, there and everywhere.

Re: Server Status: Doing fine

Postby Winterbrass » Sun Jan 24, 2010 9:36 pm

Good to have confirmation. Merci, jorb.
jorb wrote:Haven aims to be hardcore.
User avatar
Winterbrass
 
Posts: 367
Joined: Thu Jan 07, 2010 6:12 pm

Re: Server Status: Doing fine

Postby Cradlejoe » Sun Jan 24, 2010 9:40 pm

This is good to know.
At first I thought it was something to do with my internet connection but after digging round I found out it wasn't.
And good to know you guys are onto the problem.
User avatar
Cradlejoe
 
Posts: 16
Joined: Fri Jan 15, 2010 9:05 pm

Re: Server Status: Doing fine

Postby warrri » Sun Jan 24, 2010 10:35 pm

Have you checked your router settings, which could prohibit one program or pc to use more than said bandwidth?
The world I love The tears I drop To be part of The wave can't stop
Ever wonder if it's all for you
The world I love The trains I hop To be part of The wave can't stop
Come and tell me when it's time to
User avatar
warrri
 
Posts: 1033
Joined: Fri Aug 28, 2009 5:55 pm

Re: Server Status: Doing fine

Postby Rhiannon » Sun Jan 24, 2010 11:48 pm

Also, just a remote possiblity, but many service providers are very sneaky, you pay for said amount of speed and yet they have a bandwidth trip-throttle on the connection. There's even a tweak setting in my XPproBlack to override this if my isp tries to throttle my speed. Probably not what is going on, but still it's a real thing that many isps do without informing their customers.

And PS: I likes Russians..Russian guys are super smexy :mrgreen:
************************************ RAVENHOLM ************************************
***************************** IN LIEBE UND FREUNDSCHAFT *************************
************************************* Für Immer **************************************
User avatar
Rhiannon
 
Posts: 540
Joined: Wed Nov 04, 2009 3:01 am
Location: Ravenholm

Re: Server Status: Doing fine

Postby Alamarian » Sun Jan 24, 2010 11:53 pm

Rhiannon wrote:And PS: I likes Russians..Russian guys are super smexy :mrgreen:


Only the tennis players.
"The path of my life is strewn with cowpats from the devil's own satanic herd."
User avatar
Alamarian
 
Posts: 310
Joined: Mon Dec 14, 2009 11:23 pm

Re: Server Status: Doing fine

Postby Gantz » Mon Jan 25, 2010 12:35 am

4) Heirs of Rurik

Grandchildren of Lenin
User avatar
Gantz
 
Posts: 10
Joined: Sat Jan 16, 2010 1:48 am

Re: Server Status: Doing fine

Postby martinuzz » Mon Jan 25, 2010 1:32 am

Not getting my hopes up too high yet.. But it seems the lag has gone now
martinuzz
 
Posts: 381
Joined: Sat Dec 19, 2009 12:41 pm

Re: Server Status: Doing fine

Postby loftar » Mon Jan 25, 2010 2:19 am

warrri wrote:Have you checked your router settings, which could prohibit one program or pc to use more than said bandwidth?

Silence, lowlife! Of whom do you think you speak?! :)

I wouldn't use one of those cheap consumer-grade routers for anything. I always set up a Linux server to do routing. In Jorb's case, that's the same computer that runs the game server.

Rhiannon wrote:Also, just a remote possiblity, but many service providers are very sneaky, you pay for said amount of speed and yet they have a bandwidth trip-throttle on the connection

I did consider the possibility (and the related possibility that it was just Jorb's neighbors overloading the collective line by uploading too much pr0n), but was able to rule it out empirically.

However, I think I've finally found out the cause of the lag. I didn't discover it at first, because the first million times I've tried debugging it, it seems my very act of trying to debug the problem made it disappear (making it exhibit the typical features of a Heisenbug, but I'm pretty sure it's just coincidence and bad luck), so when I checked out the first few sets of debug data, I had to exclude the possibility that I initially thought would have been the problem. Having finally caught it in action, it turned out to be my first guess after all -- since all packets are sent after the main part of each game cycle (so that they can be analyzed collectively and assembled into as few physical packets as possible) they are highly temporally compressed, and at this point they simply overrun the send buffer offered by the OS. I've got a couple of ideas of different ways to fix it; I'll just have to try and choose the best one. (And verify that I'm right.)

EDIT: And just for the record; when Jorb says "Mpb/s", he means either "Mbps" or "Mb/s"; and when he says "bandwidth-spikes", he means "packet-loss spikes". ;)
"Object-oriented design is the roman numerals of computing." -- Rob Pike
User avatar
loftar
 
Posts: 8926
Joined: Fri Apr 03, 2009 7:05 am

Re: Server Status: Doing fine

Postby Lothaudus » Mon Jan 25, 2010 2:47 am

Pfft. You probably just have to jiggle the printer cable. The bytes get stuck in there sometimes.
Barry Fletcher Norwin
Lawspeaker of Crossroads - Leader of Codexia - Keeper of the Cheese
http://www.rpgcodex.net
User avatar
Lothaudus
 
Posts: 641
Joined: Fri Oct 23, 2009 12:32 pm
Location: Adelaide, Australia

Next

Return to Announcements

Who is online

Users browsing this forum: No registered users and 4 guests