Server Issues: Crashes, Lag, Restarts, Rollbacks

[jorb]

To avoid the speculation that's been going on over the state of our servers, here's the real story, all collected in one.

Any future discussion on server status will take place in *this* thread. From here on to eternity, or the next official post on the subject. Threads will not be retroactively merged, because there's a lot of BS in the other threads. I will keep the latest official server status as a sticky at all times. Keep it out of the Inn and Bugs.

1) Server is, performance wise, tagging along just fine. Hard drive, RAM and CPU are handling the recent influx of players very well with some trivial optimization of map load/save times etc.
Loftar edit: When Jorb says "tagging along just fine", he really means that it isn't currently breaking. However, it should not be construed to mean that there isn't the slightest bit of strain on it. It wouldn't manage any vast amount more players right now. If I were to guess, things might start getting a bit ugly around 300-400 simultaneous players, but maybe earlier depending on how various algorithms turn out to scale. I do have plans for optimizing various parts of the server code, mind you, but I haven't done so yet, and doing so will take a bit more time than I have right now.

2) We have had some massive bandwidth problems lately. Technically we should have 10 Mbps sending band -- we're currently sending *2.5 -- but for some reason (Which loftar is working on) we can't seem to get much more output than that. While this remains an issue, we will have the lag. We are working on this. This is a priority.
Loftar edit: As noted later on in this thread, I have a potential fix for this now, but it hasn't been verified to work yet due to too few players being logged in right now. We'll see if it works once the russians get back online.

3) We do not believe we have been DoS:ed. We're certainly not under attack. We've had two massive bandwidth-spikes, but they do not seem to be related

4) Heirs of Rurik are, as a matter of simple course, very welcome in the H&H game, and forums. We are proud to have you, please bring your friends.

Loftar can fill this in with actual tech if he wants to. I suggest he do that as marked comments entered in as edits of this entry. Over and out.

[Winterbrass]

Good to have confirmation. Merci, jorb.

[Cradlejoe]

This is good to know.
At first I thought it was something to do with my internet connection but after digging round I found out it wasn't.
And good to know you guys are onto the problem.

[warrri]

Have you checked your router settings, which could prohibit one program or pc to use more than said bandwidth?

[Rhiannon]

Also, just a remote possiblity, but many service providers are very sneaky, you pay for said amount of speed and yet they have a bandwidth trip-throttle on the connection. There's even a tweak setting in my XPproBlack to override this if my isp tries to throttle my speed. Probably not what is going on, but still it's a real thing that many isps do without informing their customers.

And PS: I likes Russians..Russian guys are super smexy

[Alamarian]

And PS: I likes Russians..Russian guys are super smexy

Only the tennis players.

[Gantz]

4) Heirs of Rurik

Grandchildren of Lenin

[martinuzz]

Not getting my hopes up too high yet.. But it seems the lag has gone now

[loftar]

Have you checked your router settings, which could prohibit one program or pc to use more than said bandwidth?

Silence, lowlife! Of whom do you think you speak?! :)

I wouldn't use one of those cheap consumer-grade routers for anything. I always set up a Linux server to do routing. In Jorb's case, that's the same computer that runs the game server.

Also, just a remote possiblity, but many service providers are very sneaky, you pay for said amount of speed and yet they have a bandwidth trip-throttle on the connection

I did consider the possibility (and the related possibility that it was just Jorb's neighbors overloading the collective line by uploading too much pr0n), but was able to rule it out empirically.

However, I think I've finally found out the cause of the lag. I didn't discover it at first, because the first million times I've tried debugging it, it seems my very act of trying to debug the problem made it disappear (making it exhibit the typical features of a Heisenbug, but I'm pretty sure it's just coincidence and bad luck), so when I checked out the first few sets of debug data, I had to exclude the possibility that I initially thought would have been the problem. Having finally caught it in action, it turned out to be my first guess after all -- since all packets are sent after the main part of each game cycle (so that they can be analyzed collectively and assembled into as few physical packets as possible) they are highly temporally compressed, and at this point they simply overrun the send buffer offered by the OS. I've got a couple of ideas of different ways to fix it; I'll just have to try and choose the best one. (And verify that I'm right.)

EDIT: And just for the record; when Jorb says "Mpb/s", he means either "Mbps" or "Mb/s"; and when he says "bandwidth-spikes", he means "packet-loss spikes". ;)

[Lothaudus]

Pfft. You probably just have to jiggle the printer cable. The bytes get stuck in there sometimes.