I just installed a new client to try and debug this. It has a new :audiobuf console command that sets the audio buffer size, to see if that is the problem. It is run like ":audiobuf 1024", for example, which sets the audio buffer to 1024 samples, which is also the default value.
Anyone who has this problem, please try setting the buffer size to something much larger, like ":audiobuf 8192" and see if that relieves the problem. If it does, please try values in between to try and find the minimum buffer size that works for you.
For the record, my system is perfectly fine with buffer sizes as small as 128 samples, which is why I thought 1024 would be more than well enough margin.
By the way, if you have this problem, please post whether you have a single-core CPU. I kinda doubt those even exist anymore, but just to make sure.