Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Announcements about major changes in Haven & Hearth.

Re: Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Postby Hrenli » Wed Feb 27, 2019 2:20 pm

loftar wrote:Neither is very likely, since they would mainly cause CPU load rather than I/O load.


I usually frown at the bots conspiracy, but just theoretically speaking login/logout would impliy I/O, am I right? I guess it is possible and could be sort of beneficial to have some bots stationed at different harvesting locations logging in and out to check for whatever they are supposed to check. And if the script in question starts to do it non-stop it might be an issue (but should be also visible and detectable)...
Too old to rock-n-roll, too young to die.
Hrenli
 
Posts: 1169
Joined: Mon Mar 21, 2016 1:39 pm

Re: Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Postby shubla » Wed Feb 27, 2019 2:21 pm

Hrenli wrote:
loftar wrote:Neither is very likely, since they would mainly cause CPU load rather than I/O load.


I usually frown at the bots conspiracy, but just theoretically speaking login/logout would impliy I/O, am I right? I guess it is possible and could be sort of beneficial to have some bots stationed at different harvesting locations logging in and out to check for whatever they are supposed to check. And if the script in question starts to do it non-stop it might be an issue (but should be also visible and detectable)...

We have no way of knowing how loftar has implemented servers, so we can only guess which operations require I/O and which ones do not.
Without knowing this it is very difficult to find the cause.
Image
I'm not sure that I have a strong argument against sketch colors - Jorb, November 2019
http://i.imgur.com/CRrirds.png?1
Join the moderated unofficial discord for the game! https://discord.gg/2TAbGj2
Purus Pasta, The Best Client
User avatar
shubla
 
Posts: 13043
Joined: Sun Nov 03, 2013 11:26 am
Location: Finland

Re: Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Postby loftar » Wed Feb 27, 2019 2:22 pm

Hrenli wrote:I usually frown at the bots conspiracy, but just theoretically speaking login/logout would impliy I/O, am I right?

Only rather minor amounts, and very certainly the kind that I would notice particularly well, so I think I'll have to deny that theory.
"Object-oriented design is the roman numerals of computing." -- Rob Pike
User avatar
loftar
 
Posts: 8926
Joined: Fri Apr 03, 2009 7:05 am

Re: Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Postby Granger » Wed Feb 27, 2019 2:22 pm

loftar wrote:
Granger wrote:My money is on it either being midges and/or bees (the patrolling ones) or the wolf AI running in circles.

Sevenless wrote:I'm gonna throw out a wildly wrong guess and say someone programmed some shitty bots that are unintentionally DDoSing the server with their requests ingame.

Neither is very likely, since they would mainly cause CPU load rather than I/O load.

Does it read or does it write? And if it is writes, what kind of data (maps, players, whatever else you persist)?
⁎ Mon Mar 22, 2010 ✝ Thu Jan 23, 2020
User avatar
Granger
 
Posts: 9263
Joined: Mon Mar 22, 2010 2:00 pm

Re: Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Postby loftar » Wed Feb 27, 2019 2:28 pm

Granger wrote:Does it read or does it write? And if it is writes, what kind of data (maps, players, whatever else you persist)?

The situation is a bit mysterious, because it's not like the system is "saturated" with I/O in some meaningful manner, from what I can tell. I'm very far from sure exactly what the problem is (it's quite hard to reproduce in an environment where I can properly debug it), but I strongly get the feeling that the a core part of the problem is sync-related. I really have no idea where in the storage stack the problem occurs, but I get the strong impression that whenever a process runs any sync/fsync/fdatasync operation, that causes all other system I/O to be congested. I'm not sure that describes the entirety of the problem either, though; I generally get the feeling that many operations that should be possible to do in the page cache block unnecessarily here and there.

Either way, the general problem is not so much about I/O bandwidth as it is about I/O latency, and that's where I expect SSDs to make an impact.
"Object-oriented design is the roman numerals of computing." -- Rob Pike
User avatar
loftar
 
Posts: 8926
Joined: Fri Apr 03, 2009 7:05 am

Re: Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Postby Granger » Wed Feb 27, 2019 2:35 pm

loftar wrote:Either way, the general problem is not so much about I/O bandwidth as it is about I/O latency, and that's where I expect SSDs to make an impact.
Let's hope you're right with this. Good luck!
⁎ Mon Mar 22, 2010 ✝ Thu Jan 23, 2020
User avatar
Granger
 
Posts: 9263
Joined: Mon Mar 22, 2010 2:00 pm

Re: Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Postby Hrenli » Wed Feb 27, 2019 2:46 pm

shubla wrote:We have no way of knowing how loftar has implemented servers, so we can only guess which operations require I/O and which ones do not.


Of course. That's why I put emphasis on theoretically speaking. But logging in would imply reading persistent data, probably loading some world chunks etc... Same with logging out...
Too old to rock-n-roll, too young to die.
Hrenli
 
Posts: 1169
Joined: Mon Mar 21, 2016 1:39 pm

Re: Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Postby rye130 » Wed Feb 27, 2019 2:54 pm

Granger wrote:
loftar wrote:Either way, the general problem is not so much about I/O bandwidth as it is about I/O latency, and that's where I expect SSDs to make an impact.
Let's hope you're right with this. Good luck!


Do you have any plans for how to pinpoint the root cause if the new hardware doesn't completely fix things?
User avatar
rye130
 
Posts: 2548
Joined: Mon Feb 01, 2010 9:41 pm

Re: Planned downtime: OS upgrade (2019-02-25 01:00 UTC)

Postby loftar » Wed Feb 27, 2019 2:57 pm

rye130 wrote:Do you have any plans for how to pinpoint the root cause if the new hardware doesn't completely fix things?

I do have some entry points to look at, but it would certainly require some investigation.
"Object-oriented design is the roman numerals of computing." -- Rob Pike
User avatar
loftar
 
Posts: 8926
Joined: Fri Apr 03, 2009 7:05 am

Previous

Return to Announcements

Who is online

Users browsing this forum: Naylok, Python-Requests [Bot], Yandex [Bot] and 16 guests