Those results in full:

1. local db test machine not available yet. We're going to nab rsw5 for a while this afternoon and use that instead.

2. according to the explain stats, we are faster with the new approach than we were with the tried-and-tested mark links approach (900 versus 1700). Unfortunately, that's not borne out by the tests, which put us about 10 times slower (1.2 seconds versus .12 seconds per query).

I have a theory about this, which is that our new approach requires much more CPU activity than the old one and the explain stats are geared towards IO cost. That means that if the machine is heavily loaded (as our server is) then the new approach will suffer much more than the original. Steve's checking out whether I'm talking rubbish, which is always a possibility. A small flame of hope survives that it's all going to rock when we try it on a responsive machine.

3a. Tried. Discarded. It ramps the cost up several orders of magnitude.

3b. Tried. Incorporated. Raises the cost somewhat (900 to 1050) but is much more scalable and will speed imports.

The presence of the new indexes is slowing imports, so I'm dropping them while the imports go through. I think we'll have to build those steps into the import process.

Matt's seeing memory usage rise continually as tomcat handles requests, so I'm getting OptimizeIt connected to the process to see what's going on. Is it us? Is it tomcat? Enquiring minds want to know.

In the meantime, Steve's verifying the stats on our link retrieval, preparing a script to add the indexes and getting ready to test on w5.

T
- posted by Tim Bruce @ 2:07 PM