Lag. Again. Seriously.

[warrri]

Ok seriously, the lag is back and getting worse each day. Its not as bad as before the ram upgrade yet, but its definitely aiming to beat it.
In fact it was only really well like one week long, then it started to lag again. It's really annoying and getting unbearable already. Just do something please. It cant be that hard to write proper pathfinding algorithms for cow and sheep herds. We're doing that in our second semester already, though i didnt pay attention because, i have to admit, it is very boring. But comeon, you will be able to do that!

[loftar]

There is nothing that indicates that there's any lag that is caused by cow or sheep herds, however. :) Actually, I was worried about how much more CPU the domesticated animals might use, but so far it hasn't made any difference at all that I've been able to measure, which has made me quite glad.

There are a few categories of lag sources that I have been able to identify so far:

• For some reason (I'm not sure why yet) a larger amount of map has been loaded into the server as of late. That isn't a problem in itself, but it leads to a higher frequency of saves of map data, which drains pretty much I/O bandwidth. I have a few theories on how to fix it, but it requires closer examination and profiling. Also, some parts of the map require unproprtionately much CPU time to save, which needs some profiling to determine the exact cause.
• I don't know really why, but for some reason, the server's I/O (even when not saving maps) has been more strained lately, and that means that map loads take longer than they should. Often, it just takes a few milliseconds of actual CPU time to load a map, but several tens of a second to load it from disk, which is bad. One way to fix it would be to manage the server's I/O bandwidth better (for instance, by putting map data on a dedicated hard drive). Another way to fix it might be to predict where map is going to be loaded soon and try to prewarm the block cache by reading the file data asynchronously in advance.
• There's a weird problem that happens sometimes when people log in, which makes the server hang for anywhere from 1-10 seconds. It seems to have started happending after I wrote the new account server, but it has been very hard to catch it in action (it exhibits Heisenbug characteristics :P), so I haven't been able to figure out what causes it.
• I recently discovered that the filesystem on the server was formatted without the ext2+ dir_index filesystem option, which is quite bad for the performance of some parts of loading and saving. Fixing that requires reindexing the filesystem and would require quite a bit of server downtime. I'll probably fix it soon anyway, since I got my hands on some new hardware to run the server on (it has a dual core CPU, which will also fix some minor lag that happens when some other process (usually Apache) preempts the Haven server).
• Map loading sometimes takes too much CPU time for its own good. It isn't exactly very weird, since some parts of the map contains many hundreds of thousands of objects (like palisade wall segments) that need to be created in less than 60 ms, but it's bad nevertheless. I've done some preliminary profiling on it, and it seems that there are some parts of the related code that can be optimized very much with comparatively little effort, but more experimentation is necessary. There are other parts that definitely should and probably can be optimized that would yield massive gains, but where there is quite a bit more work involved. I'd really love to put my hands on it, but my time is, unfortunately, limited.
• An interesting and probably very fruitful, but very massive, project would be to try and parallelize the Haven server, so that maps can truly be saved and loaded asynchronously (and also so that, in the future, more simultaneous users can be supported by adding more CPUs). Trying to devise a reasonable locking scheme for the server is very challenging, however. (Now, I'm not one to decline such a fun challenge, of course, but there are scores of other things that need to done before I get there.)
• Generally, it can be said that the Haven server exhibits many of the problems related to real-time programming (since each cycle has a hard limit on how much time it is allowed to take), which is something that I've had very limited experience with previously -- especially in relation to how it interacts with the rest of the operating system -- so I'm learning as I go. Many problems have already been fixed in the past, and I've developed a number of new techniques to analyze the problems as they show up, so things are becoming better.

[Alamarian]

• I don't know really why, but for some reason, the server's I/O (even when not saving maps) has been more strained lately, and that means that map loads take longer than they should. Often, it just takes a few milliseconds of actual CPU time to load a map, but several tens of a second to load it from disk, which is bad. One way to fix it would be to manage the server's I/O bandwidth better (for instance, by putting map data on a dedicated hard drive). Another way to fix it might be to predict where map is going to be loaded soon and try to prewarm the block cache by reading the file data asynchronously in advance.

If you ever have money to spare and the map data will fit on one, you might consider a sold state hard drive. The speed is massive, even if the storage size is pathetic.

[loftar]

If you ever have money to spare and the map data will fit on one, you might consider a sold state hard drive. The speed is massive, even if the storage size is pathetic.

It certainly fits -- the entire server filesystem is currently 30 GB, and the vast majority of that is actually taken up by the minimap images, which are only of client interest. :)

However, I have been wondering about those SSDs. I know that they lack seek time, of course, which is great (but not very relevant for the map data issues); but I've gotten the impression that Flash lags behind hard drives in terms of pure bandwidth. Are there any good benchmarks on those things?

[Alamarian]

However, I have been wondering about those SSDs. I know that they lack seek time, of course, which is great (but not very relevant for the map data issues); but I've gotten the impression that Flash lags behind hard drives in terms of pure bandwidth. Are there any good benchmarks on those things?

The Intel X-25E is doing a lot to change that idea. The I/O is through the roof. Sequential writing is inferior to, say, a Seagate Cheetah (15,000 RPM) but the X25E's I/O numbers are great. For a whole bunch of graphs and benchmarks, see http://www.tomshardware.com/reviews/int ... ,2158.html