Lagspike saga

[loftar]

Did you do the AB BA switch mentioned?

No, I don't have physical access to the server, but I have to admit I don't really see the point even if I did have. What are you proposing would happen if they were physically switched between slots?

[loftar]

Actually, speaking of swapping drives, it does appear that what has happened now is that the drives have in fact switched in terms of which one is causing problems. I guess that rules out hardware issues. (Unfortunately, since that would have been an easy fix if it had been true.)

[Lord_of_War]

https://en.wikipedia.org/wiki/Muntzing If the problem drive switched try without RAID.

[Ferinex]

https://en.wikipedia.org/wiki/Muntzing If the problem drive switched try without RAID.

If data redundancy is considered a non-functional requirement, then 2 drives in RAID is already the minimum required for the system. I think you've shed light on a different problem though, which is that the prod environment doesn't closely resemble his test environment, so he is not able to do this sort of experimentation without causing downtime. Renting a test environment that matches the prod environment of course is one thing for which they probably need funding

[MagicManICT]

https://en.wikipedia.org/wiki/Muntzing If the problem drive switched try without RAID.

i'm not a strong Linux guy (yet), but doesn't this require rebuilding the drive structure? It at least does in Windows and can take some time, and not likely doable with the system in a running state.

[TheNater]

Did you do the AB BA switch mentioned?

No, I don't have physical access to the server, but I have to admit I don't really see the point even if I did have. What are you proposing would happen if they were physically switched between slots?

The AB-BA swap would help determine whether it was a software or hardware issue like you mentioned here:

"There are two SSD's.....One of these is acting strangely: Monitoring the I/O to it, it exhibits periods where the drive just seems to be "idle" ------- The question, then, is what causes this, and I'm currently trying to figure out if it's a hardware or software issue."

I mentioned this since it's an easy thing to check if the server is in front of you (i.e. physical access). Since it's in a datacenter and you're not present, not so much. It's just something I would try after the update if I was in front of the servers.

[loftar]

The AB-BA swap would help determine whether it was a software or hardware issue like you mentioned here:

I don't really see how it would determine that. It seems the only thing it might determine would be if there were something wrong with the physical PCIe connection, rather than the drive.

[TheNater]

Did you do the AB BA switch mentioned?

No, I don't have physical access to the server, but I have to admit I don't really see the point even if I did have. What are you proposing would happen if they were physically switched between slots?

The AB-BA swap would help determine whether it was a software or hardware issue like you mentioned here:

"There are two SSD's.....One of these is acting strangely: Monitoring the I/O to it, it exhibits periods where the drive just seems to be "idle" ------- The question, then, is what causes this, and I'm currently trying to figure out if it's a hardware or software issue."

I mentioned this since it's an easy thing to check if the server is in front of you (i.e. physical access). Since it's in a datacenter and you're not present, not so much. It's just something I would try after the update if I was in front of the servers.

Basically, if you swapped the drives and the problem follows the 'drive' then it's hardware. If it stays the same, it's going to be a software issue.

[TheNater]

The AB-BA swap would help determine whether it was a software or hardware issue like you mentioned here:

I don't really see how it would determine that. It seems the only thing it might determine would be if there were something wrong with the physical PCIe connection, rather than the drive.

Sorry if I double-posted. You are correct, but I still think it would be beneficial information to have. It's just something I would do in my regular troubleshooting if I was right there in front of the equipment. It's just one of those "why not" things if you're in front of the server.

EDIT: Just posting in case people cant read - THE SERVER IS NOT IN FRONT OF JORB OR LOFTAR TO MAKE CHANGES.

[Lord_of_War]

There are diagnostic tests for drives done by placing an artificial load on them. However if the issue is in the software stack it won't find anything. Could redundancy be done asynchronously with COW?