A review of the markless link project and an action plan:

1. queries are averaging .7 seconds, but variability is high and some queries are unacceptably slow. Unfortunately, the office db server is busy with imports and so we can't get an accurate idea of the real-world speed. Would be good to test on production hardware.

2. In lieu of spare hardware, we can compare speeds by checking the "explain" output from psql on our original and new queries. This will give us an indication of expected page reads per query (903..903 for the new query right now).

3. We can alter the sql in a couple of ways;

a) we consult heading_entry_to_entry three times (heading, fan, reverse fan). Can we speed things up by refactoring the query and only scanning this once? This would sacrifice the ability to vary link strengths independently for these linkers, but that's acceptable because they're both fairly dumb and should be marked low.

b) we could skip the linkerx_entry_to_entry tables entirely and go back to the source tables for the query. For example, calculate the heading links on-the-fly from heading_entry_to_phrase. This seems intuitively less efficient but worth testing. It would be a big win in agility.

That's the plan for today, then. Steve's looking at 2 and 3a, I'm taking 1 and 3b.

Matt's preparing our new db machine to test the tomcat configuration and I'll take advantage of that in a spare minute to test 1. We're going with postgres 7.4, which I hear is substantially faster than our current 7.3 install.

On to 3b in the meantime.

T
- posted by Tim Bruce @ 10:43 AM