Lagspike saga

[loftar]

I realized a couple of days ago that I could use the 32 GB swap partition that I had allocated specifically to the NVME drives to debug this. Since the swap isn't normally in use, I deactivated it and trimmed those entire partitions. Immediately afterward, the lagspikes decreased substantially, but since then they seem to have been coming back again, so it wasn't exactly a resounding success.

Having looked at more I/O traces since then, however, I just can't help but get the feeling that it should be TRIM-related, because the only I/O operations that seem to be getting high latency are writes, and at the times when they happen, large batches of writes are often completed at once, after they have experienced high latency together. Not sure what to make of that, but it's not like it didn't seem to do "something", and it's not like it couldn't be that 32 GB is just not enough (though that analysis also does sound quite optimistic). Perhaps I should try to reserve more unused space on the drives and try again. That will take some downtime, however, so I think I'll ponder it a bit more.

[Granger]

As far as I understood the bcache documentation you should be able to completely detach (remove) a cache device (performance will be *meh* then as they'll be written back to the base device which then will take over service) so you should be able to repartition the NVMe's on-the-fly.

[loftar]

As far as I understood the bcache documentation you should be able to completely detach (remove) a cache device (performance will be *meh* then as they'll be written back to the base device which then will take over service) so you should be able to repartition the NVMe's on-the-fly.

True enough. I was thinking that performance might be bad enough that I might as well shut down, but I guess there's no reason to draw that conclusion in advance.

[Granger]

As far as I understood the bcache documentation you should be able to completely detach (remove) a cache device (performance will be *meh* then as they'll be written back to the base device which then will take over service) so you should be able to repartition the NVMe's on-the-fly.

True enough. I was thinking that performance might be bad enough that I might as well shut down, but I guess there's no reason to draw that conclusion in advance.

On the other hand: the last worlds ran on the old server quite nicely, that one only had HDDs...

[Grog]

The lagspikes now are insane.

[Fierce_Deity]

Seems to be especially bad right now.

[Granger]

As far as I understood the bcache documentation you should be able to completely detach (remove) a cache device (performance will be *meh* then as they'll be written back to the base device which then will take over service) so you should be able to repartition the NVMe's on-the-fly.

True enough. I was thinking that performance might be bad enough that I might as well shut down, but I guess there's no reason to draw that conclusion in advance.

As a polite reminder: At what point in time do you plan to tackle this?

[jordancoles]

Getting laggy peaks every 3 seconds or so, had to log off because it's not playable

[Sarge]

It's been particularly bad since yesterday.

[Omnipotent]

+1 to temporary shutdown for possible fix. I don't mind waiting even if its down for an entire day.

The last 48 hours have been particularly bad as has been mentioned by Sarge & Coles. Not sure if its just the bots from major factions or what, but it can be quite unpleasant at times of severe lag. Anything would be better than nothing.