Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

[Hrenli]

Neither is very likely, since they would mainly cause CPU load rather than I/O load.

I usually frown at the bots conspiracy, but just theoretically speaking login/logout would impliy I/O, am I right? I guess it is possible and could be sort of beneficial to have some bots stationed at different harvesting locations logging in and out to check for whatever they are supposed to check. And if the script in question starts to do it non-stop it might be an issue (but should be also visible and detectable)...

[shubla]

Neither is very likely, since they would mainly cause CPU load rather than I/O load.

I usually frown at the bots conspiracy, but just theoretically speaking login/logout would impliy I/O, am I right? I guess it is possible and could be sort of beneficial to have some bots stationed at different harvesting locations logging in and out to check for whatever they are supposed to check. And if the script in question starts to do it non-stop it might be an issue (but should be also visible and detectable)...

We have no way of knowing how loftar has implemented servers, so we can only guess which operations require I/O and which ones do not.
Without knowing this it is very difficult to find the cause.

[loftar]

I usually frown at the bots conspiracy, but just theoretically speaking login/logout would impliy I/O, am I right?

Only rather minor amounts, and very certainly the kind that I would notice particularly well, so I think I'll have to deny that theory.

[Granger]

My money is on it either being midges and/or bees (the patrolling ones) or the wolf AI running in circles.

I'm gonna throw out a wildly wrong guess and say someone programmed some shitty bots that are unintentionally DDoSing the server with their requests ingame.

Neither is very likely, since they would mainly cause CPU load rather than I/O load.

Does it read or does it write? And if it is writes, what kind of data (maps, players, whatever else you persist)?

[loftar]

Does it read or does it write? And if it is writes, what kind of data (maps, players, whatever else you persist)?

The situation is a bit mysterious, because it's not like the system is "saturated" with I/O in some meaningful manner, from what I can tell. I'm very far from sure exactly what the problem is (it's quite hard to reproduce in an environment where I can properly debug it), but I strongly get the feeling that the a core part of the problem is sync-related. I really have no idea where in the storage stack the problem occurs, but I get the strong impression that whenever a process runs any sync/fsync/fdatasync operation, that causes all other system I/O to be congested. I'm not sure that describes the entirety of the problem either, though; I generally get the feeling that many operations that should be possible to do in the page cache block unnecessarily here and there.

Either way, the general problem is not so much about I/O bandwidth as it is about I/O latency, and that's where I expect SSDs to make an impact.

[Granger]

Either way, the general problem is not so much about I/O bandwidth as it is about I/O latency, and that's where I expect SSDs to make an impact.

Let's hope you're right with this. Good luck!

[Hrenli]

We have no way of knowing how loftar has implemented servers, so we can only guess which operations require I/O and which ones do not.

Of course. That's why I put emphasis on theoretically speaking. But logging in would imply reading persistent data, probably loading some world chunks etc... Same with logging out...

[rye130]

Either way, the general problem is not so much about I/O bandwidth as it is about I/O latency, and that's where I expect SSDs to make an impact.

Let's hope you're right with this. Good luck!

Do you have any plans for how to pinpoint the root cause if the new hardware doesn't completely fix things?

[loftar]

Do you have any plans for how to pinpoint the root cause if the new hardware doesn't completely fix things?

I do have some entry points to look at, but it would certainly require some investigation.