Planned downtime: Filesystem test (2019-04-11 22:00 UTC)

[shubla]

I estimate half an hour.

Haha, well, that was funny. The I/O on the server is pretty fubar. When I do things like this, I'm impressed it doesn't lag more in normal operation.

Next time say that it will take 6 hours so then it will probably take like 30 minutes?

[Oddity]

[loftar]

Finally, the copy back is done. Will start the game server back up in less than five minutes.

The copy process ran at 4 MB/s. That is quite ridiculous.

[WaitWhat4355]

Ah, the glorious power of SSDs. The technology of today is amazing.

[loftar]

For the record, my main working theory right now is that the SSDs aren't overprovisioned enough, but it would be nice to have some idea whether that's likely to be true or not. The SSDs are 512 GB large, and I've provisioned off 32 GB of unused space on them for the FTL to do its stuff. If there's anyone in the know around here, I'd really appreciate if you could riddle me whether you think 32 GB is ridiculously little and of course it should be more than that, or whether 32 GB should be more than well enough and the problem obviously has to be elsewhere.

Ah, the glorious power of SSDs. The technology of today is amazing.

[WaitWhat4355]

From what I know, most SSDs already have some over provisioning that you can't touch but how much it is varies from manufacturer to manufacturer. If it's over 75% of the capacity it shows, then you should start getting some performance drops and it could be a lack of space to shift things around being the issue. How full are they right now?

Also, I was trying to make a joke that amazing pieces of tech that everyone loves can have failures for no apparent reason.

[loftar]

How full are they right now?

Well, the space that is used is pretty much 100% used since, as mentioned previously in the thread, they're being used for bcache. So since only the previously mentioned 32 GB are unused, that would make them 93.75% full.

From what I know, most SSDs already have some over provisioning that you can't touch but how much it is varies from manufacturer to manufacturer.

I don't believe these ones have any of that.

If it's over 75% of the capacity it shows

Is that a somewhat firm statement? Are you saying that I should effectively reserve 128 GB of unused space on them?

[WaitWhat4355]

If it's over 75% of the capacity it shows

Is that a somewhat firm statement? Are you saying that I should effectively reserve 128 GB of unused space on them?

I'd imagine this was more or less the starting point of when SSDs started slowing down to a barely noticeable degree, but as it went further over it'd become more and more noticeable, on the smaller drives at least. This was how it was for SSDs from a few years ago, so they might've changed it for more recent ones.

How full are they right now?

Well, the space that is used is pretty much 100% used since, as mentioned previously in the thread, they're being used for bcache.

Bcache is where it stores the most frequently used files from the HDD for faster access and puts the files sent for storage before being flushed to the HDD, right? Do you know if it it forces all data sent to storage through it, or sometimes lets it bypass directly to HDD if needed? If it's trying to make everything go through the bcache SSDs as the only inlet to the storage and they are at a constant 100% with 32GB only for overprovision, then it could be that the SSDs are currently too slow at moving things around in them with the lack of space and overhead.

From what I know, most SSDs already have some over provisioning that you can't touch but how much it is varies from manufacturer to manufacturer.

I don't believe these ones have any of that.

I thought all SSDs had this, and HDDs included? Though, HDDs had a much smaller % of overprovisioning due to how it had only small blocks on the disk, could be moved anywhere, and had indefinite rewrites. SSD manufacturers never actually tell you how much is over provisioned.

[TheNater]

Anddd she's down.

[WaitWhat4355]

Anddd she's down.

The game is running fine for me, but it did have a minute long lag spike as my guy hammered a cellar the whole time without progress.