Tweaked atlas code so that it can use a different database from that which the direct qt picks up. Atlas demo is now live on xplus.

Extended drop-down so it features 2 percent of the gazetteer info, drawn from round the world and sorted by country and place name.

Looks good!

Next bits: see if we can tighten up the projection maths any further, join panel-boundary tiles up and start work on an rml test file. Might all have to wait until next Tuesday.

T
- posted by Tim Bruce @ 5:07 PM 0 comments

 
Found lots of web resources about the Universal Transverse Mercator projection (UMT), a beast which splits the globe into 60 sections and chooses a new meridian for each one, minimizing the distortion as far as is practical. Hmm. Tried a couple of online calculators but they don't seem to line up with the gazetteer information either. Definitely know more about map projections than I did this time last week!

Tweaked the TargetDisplay code so it's working better and re-inserted the drop-down for French places. Alignment seems very good over France, now to broaden the scope of the drop down so we've got a global selection.

Map tile generation coming along well.

T
- posted by Tim Bruce @ 3:18 PM 0 comments

 
Got the basic highlight mechanism into place.

Tiles are getting rendered through a filter which works out where the target is in relation to the tile extents and draws a blob (currently a shadowed yellow rectangle) on the tile before streaming the image out as a png.

Uses the new javax.imageio package (well, new to me anyway).

It's really showing up the errors in our projection - near the equator the blob is too high, near the pole the blob is too low. I chose our constant so that London lines up, but it would be good to find out exactly what transformation was used for the map. Longitude is dead on.

T
- posted by Tim Bruce @ 5:35 PM 0 comments

 
Image manipulation was taking a very long time on goring and disrupting an import which was running alongside. Now stopped.

Combination of the panels before slicing into tiles turned out to be impractical - the files get unmanageably large very quickly.

Taken alternate approach and adapted Geoff's image slicer so that it takes the mosaic of panels into account when creating tiles, placing boundaries where appropriate and iterating through the panels. You end up with lots of files called tile_rxc.png where r=row and c=col. For tiles which are split between panels, we create multiple files with markers in the filename to indicate which bit of the tile is in each file (eg tile_2x8lt.png is the top-left corner of row 2 col 8). Will stick these together appropriately once processing is complete.

Running the code on our best spare machine, the new office db server. It's not yet getting used for the database. Has 2 xeon processors, so running 4 copies of the script in parallel, each working on a different row of panels. Much faster than goring! Will have to go back and process the rest of row 1 once one of the other processes has finished.

Re-ran the tile generator app to populate the database with our complete set of tile details.

Hooked up a symbolic link under mediatest so that the atlas test page has access to the growing collection of tiles. Found a couple of islands in the pacific so far, so the coordinate system is at least roughly working.

T


- posted by Tim Bruce @ 2:31 PM 0 comments

 
Added a static drop-down of all the French gazetteer locations on the atlas page. This makes for easier testing!

Next to add a marker to the map to guide you to your chosen target.

goring's working on the image manipulation in the background.

T
- posted by Tim Bruce @ 11:31 AM 0 comments

 
Working on the image tiles to back the atlas.

Because our 200x200 pixel tiles won't fit neatly into the supplied images, I'm trying to combine the existing images into as large sections as possible to minimize fiddly alignment work at the boundaries. There are 6 rows of 8 tiles, and I'm using imagemagick to combine them into just 6 wide images.

Then into PNGs to see if they're easier to handle than TIFs (ie. are they any smaller?).

Then combine as many rows as possible before splitting into 200s.

T
- posted by Tim Bruce @ 11:02 AM 0 comments

 
Added new features to our list element. You can now specify a colour for the item marker, and nested lists are supported with automatic indentation.

T
- posted by Tim Bruce @ 4:18 PM 0 comments

 
Extended dao to retrieve a 3x3 tile area around your focus.

Hooked up to the atlas test page which puts the available tiles into a table.

Seems to work OK as we tool around Europe - need to import more data so we can see how the distortion affects calculations around the globe.

T
- posted by Tim Bruce @ 5:30 PM 0 comments

 
Finished projection-aware tile importer and regenerated tile info.

Hooked atlas demo jsp up to atlas dao and added a lon/lat form so the user can request locations.

Seems to work OK, need to extend dao so that we get more than one tile.

T
- posted by Tim Bruce @ 4:52 PM 0 comments

 
Added architecture for switchable atlas-specific projections, along with a rough projection for the current data.

Adding tile importer which uses the projection to calculate lat/lon for each tile.

T
- posted by Tim Bruce @ 5:25 PM 0 comments

 
The mercator transformation gives us a better match to the grid coordinates, but it's still not quite linear. Will press ahead using this transformation for the moment to see where it gets us.

We'll be keeping all our internal processing in lon/lat coordinates so we don't have to worry about projection issues anywhere except the user interface. It does look as if we may have to store separate transformations for each atlas we use.

T
- posted by Tim Bruce @ 2:03 PM 0 comments

 
Yesterday I added a retrieval method to the AtlasDAO which goes from a LonLat and an atlas id to a matching tile. Wrapped it up into a test application.

Unfortunately, the coordinates are way off.

On staring at the tile extents file, it became apparent that it's not expressed in degrees after all. Bah! Tried to figure out the actual units used.

The longitude scale runs from -2003772600 to 2003772600.
Degree range is -180 to 180.
=> 1 degree of longitude = 11132070 units. Comparing coordinates from both systems in the gazetteer, this seems to be a constant factor.

Latitude is trickier. The conversion factor varies as latitude changes.

I think the map coordinates may be using the Mercator projection.

Have set up a latitude to mercator converter, now testing data.

T

- posted by Tim Bruce @ 11:13 AM 0 comments

 
Modified db structure stlightly, adding a projection table so we can handle atlases in formats other than Mercator.

Created value objects to represent AtlasTile, AtlasView.

Created AtlasDAO to handle the data access.

Wrote test script which uses AtlasDAO to store tile details in the db. Applied it to the test dataset so we've got some real data in the db (db_dev at this stage).

Tomorrow to hook up the renderer so we can feed in a LonLat and obtain a view of the atlas.

T
- posted by Tim Bruce @ 5:16 PM 0 comments

 
Created basic database structure to track atlases, atlas types and atlas tiles.

T
- posted by Tim Bruce @ 4:58 PM 0 comments

 
Set up a test page for the atlas at xplus/atlas.jsp

The page is very dumb at the moment - it knows nothing about maps, entries or anything else. It stitches together a 3x3 grid of tiles so we can see how the final map will look and allows you to navigate around tile-by-tile in our eight directions.

The size of the map feels OK and the tiles join up without visible seams. Hurrah!

Next to add lon / lat knowledge to the system.

T
- posted by Tim Bruce @ 12:56 PM 0 comments

 
Tweaked the metasearch boost factor from 7 to 30, which makes it fairly hardcore. This is in response to a poor image search noted by Adam.

Up on xplus for testing.

T
- posted by Tim Bruce @ 12:08 PM 0 comments

 
Updated the short description of Geography to remove reference to geology.

T
- posted by Tim Bruce @ 11:23 AM 0 comments

 
Added a new feature to the mapper - the last node which the user selected is painted a different colour. This applies both to entry views and node expansion.

Colour is currently 255, 64, 64 but is up for discussion.

T
- posted by Tim Bruce @ 10:56 AM 0 comments

 
Modified admin system so that entities are maintained in the review-edit-submit procedure (ie they appear correctly in the text boxes).

T
- posted by Tim Bruce @ 5:45 PM 0 comments

 
Spot-fixed some high-ascii in stats pages for clients whose names should include entities and republished.

T
- posted by Tim Bruce @ 4:59 PM 0 comments

 
Tile generation is complete for the test dataset, and I'm running a batch conversion to gif files.

The tiles look to be a useful size, and most of them seem to contain at least one feature for which we can generate an entry (gazeteer permitting).

Started roughing out the atlas rendering system. Added a LonLat class and unit tests for it.

T
- posted by Tim Bruce @ 4:12 PM 0 comments

 
Had a conversation with Carl and we've refined the navigation plan, ie the question of "what happens when you click the arrow for west?"

1. For the purposes of this project, the atlas data will always be associated with the same book. This means that we can link each tile which contains a gazeteer feature to an entry.

2. On the original plan, you'd only be able to hit "west" if there was a feature in the square to the centre left of your grid. The system would return the entry for that feature, with the appropriate square in the centre of your new grid. If there was no feature in the square, there could be no asssociated entry and so the west arrow would be greyed out.

3. The new plan never greys out the arrow, but when you hit it you may "jump" across the map more than one square - perhaps even across the Pacific. We'll take you to the next square in that direction which contains a feature (and therefore has an entry). All this can be pre-calculated, and the new system makes it much easier to navigate from one shore of a lake to another or over a mountain.

T

- posted by Tim Bruce @ 12:01 PM 0 comments

 
The tile generation is just about complete - the routine is following the original file format and kicking out tiff files. Will batch-convert them to gif once they're all there.

Generated a list of tile extents so we can map from tiles back to real-world coordinates. This raises fly no. 2 - I'm using a linear mapping from pixels to lat/lon, which is probably inappropriate for this map. I reckon it's in Mercator but will have to check. Each set of atlas data will have to have its projection noted in the db so that we can draw overlays accurately. For the moment, I'll work on the linear system to get the big picture up, but keep in mind that we'll need to introduce corrections later on.

T
- posted by Tim Bruce @ 11:06 AM 0 comments

 
First fly in the ointment - we'll have to tread carefully when it comes to boundaries between the original images. There's no guarantee that they will align conveniently with our 200x200 grid, so we'll have to do a bit of judicious joinery to ensure the seams are invisible.

Tackle that later. For the test we're only using one of the original tiles.

I'm using Geoff's image slicing script to generate the tiles, but due to the size of the original image it's going to be a lengthy process.

T
- posted by Tim Bruce @ 4:57 PM 0 comments

 
Translated the excel file which lists tile areas into csv format so we can work with it.

We're going to be using original tile13 for the development tests - Western Europe.

First step is to slice it up. How large should the tiles be? Going to start with 200 pixels and see how that works out. That means that the atlas will be 600px square on entry pages.

T
- posted by Tim Bruce @ 3:57 PM 0 comments

 
Things to do:

1. Split image data into tiles
2. Store tile filenames in db, alongside details of area covered, adjacent tiles, scale and mapping type (vector, raster, roadmap, outdoor leisure, etc).
3. DB handling which accepts a location (lat/lon), scale and map type and returns HTML which renders the atlas at that location.
4. RML extension to invoke the AtlasRenderer.
5. Reverse mapping in db from each tile to an entry.
6. Navigation using atlas arrows.
7. Scale navigation.

More will occur on the way, but I reckon that's enough to be going on with.

T
- posted by Tim Bruce @ 3:03 PM 0 comments

 
Right then: the atlas.

First stab at how it's going to work.

The new feature will allow us to display a map of any part of the world for which we have map data. It is anticipated that we might acquire a variety of map datasets which cover different regions of the world at varying levels of detail. The user's view of the map will include links to scroll the map in any of eight directions, as well as zooming in or out (data permitting).

To allow integration of the atlas with as broad a range of data as possible, we will split handling of the graphical data from the gazetteer information. The latter will be processed through the standard RML / import route to give us a new volume. RML will acquire an extension which allows it to reference a map location. When the RML is rendered, that reference will be replaced by an image which highlights the specified location at a particular scale.

To distinguish this feature from the research mapper, I'll try to stick to the term Atlas for the new system we're building!

T
- posted by Tim Bruce @ 2:20 PM 0 comments

 
Added brackets to the cluster count to indicate whether the map is still sorting.

T
- posted by Tim Bruce @ 12:28 PM 0 comments

 
Added a cluster count to the dashboard.

Kicked off sql import on rsw1.

T
- posted by Tim Bruce @ 5:46 PM 0 comments

 
Moved the frozen-cloud sorting out of MapPanel and into our jdk118 Collection util class. Replaced the temporary bubblesort with a quicksort - I don't think we'll ever be tackling large enough clouds for the difference to be noticeable, but you never know...

We're not going to pursue the "last node clicked" indicator.

Carl's assembled a basic design for the large key. Once that's back from the designers and we've done some more road testing, the v2 applet is ready to go.

On a different subject, tested rss newsfeeds.

T
- posted by Tim Bruce @ 5:37 PM 0 comments

 
Implemented a version of the mapper in which point labels extend beyond the zoom window.

Hmm. Not sure I like the aesthetics, but it could be more practical!

Playing with it.

T
- posted by Tim Bruce @ 3:34 PM 0 comments

 
The next point to fix (and this might be a biggie) is this: what should happen to node labels in zoom mode?

At the moment, we display them and clip them to the zoom lens boundary. I'm ok with that, as I think it reinforces the sensation that you're looking through a physical lens and magnifying what's beneath. I think others had other preferences, so I'll ask around this afternoon and see if we can decide what needs to be done.

T
- posted by Tim Bruce @ 12:53 PM 0 comments

 
Fixed point 1 - the zoom rectangle now obscures the key if required.

We're going to need an image for the lage key before release, unless we drop it.

T
- posted by Tim Bruce @ 12:48 PM 0 comments

 
Fixed point 2 - no large key is displayed in zoom mode.

T
- posted by Tim Bruce @ 11:08 AM 0 comments

 
Fixed point 4 - the map goes direct to entry.jsp rather than entry_cluster.jsp. Not a lot of work, that one!

T
- posted by Tim Bruce @ 10:48 AM 0 comments

 
Improved the frozen cloud feature so that the highlight starts from your current position, rather than the highest node vertically.

T
- posted by Tim Bruce @ 10:40 AM 0 comments

 
Added handling of the return key to the mapper - once you've frozen the cloud you don't need the mouse to show or expand nodes.

T
- posted by Tim Bruce @ 5:41 PM 0 comments

 
Taken on 6 first - cycling through label cloud.

Unfortunately, mousewheel support doesn't extend as far back as jdk1.8 so that's out. Alternative, for general consideration, is the space bar.

Implemented this and put it up at xplus/research/update.jsp

You've got to ensure that the applet's got input focus, then move into a label cloud and hit space.

The visible labels are now "frozen" - they move with their associated nodes, but they remain visible even if they drift away from the cursor.
Each subsequent hit on the space bar moves the highlight through the frozen cloud from top to bottom.
A mouse click acts on the highlighted node (to view or expand it).

The cloud is unfrozen as soon as the mouse is moved.

T
- posted by Tim Bruce @ 4:41 PM 0 comments

 
Mapper things to do:

1. Resolve zoom/key z-order
2. Suppress large key in zoom mode
3. Resolve label cloud in zoom mode
4. Don't display cluster entries on entry page
5. Indicate which node was last clicked
6. Use mousewheel to cycle active node through label cloud nodes
7. Test zoom on all platforms

T
- posted by Tim Bruce @ 3:43 PM 0 comments

 
Primed w1 with the February release, and updated the jserv config to include the new charting jars. Tested with the superceded cia table to check everything works. The cluster update is now in progress.

Modified parser to display the new audio.gif rather than speaker.jpg - this adds transparency to work better against our buff background option. Will go up in next jar update.

Added audio to the list of meta-search options. Will go up in next page update.

Assembling mapper test suite and list of work remaining.

T
- posted by Tim Bruce @ 2:33 PM 0 comments

 
The link to "your search results" now remembers your exact search specification, so the page you obtain really is the last one you looked at. Previously, we would lose your volume/subject selection if there was one.

T
- posted by Tim Bruce @ 4:19 PM 0 comments

 
Fixed extract generation for incoming text which contains no hits. This can arise if there's a hit in the pronunciation text, which is not displayed in results lists and so means there's no hit in the displayed result.

Previously, we would display the entire entry text in the result list. Now we trim it to the first n characters, as if it had a hit at the very start.

T
- posted by Tim Bruce @ 12:16 PM 0 comments

 
Results pages now have access to the meta-info for every result: In particular, you can check to see whether the result contains audio.

Inserted example code into results_presentation.inc to illustrate how this could be used.

T
- posted by Tim Bruce @ 10:33 AM 0 comments

 
Today is release day for the new jar, but w1 is still running an sql import and won't be finished soon. In the meantime, off to other things. I'm going to finish as many bitty projects today as I can before getting into the mapper again tomorrow.

T
- posted by Tim Bruce @ 10:24 AM 0 comments

 
Tidied desk!

T
- posted by Tim Bruce @ 5:29 PM 0 comments

 
Updated the import jar so that we get the improved dtable indexing on the cia re-import.

T
- posted by Tim Bruce @ 4:02 PM 0 comments

 
Codebase passed all junit tests and jsitetest. Tagged as "blackfriars" and deployed on office server.

T
- posted by Tim Bruce @ 3:00 PM 0 comments

 
Tweaked ContentDAO so that ArrayLists are allocated with the correct length from the start, for efficiency.

Other than that, the profiler's not showing much we can shave off the performance. Most time and object count goes into the database communication, and that's only doing the essentials now. If we really want to push it then we could look at caching the contents of the tables as well as their invariants, but I don't think it's worth it at this stage.

Final testing this afternoon, release tomorrow and re-import of the book. Then make the cia book available to all xrefer testers on Wednesday before final release this/next week.

...and then on to a different project!

T
- posted by Tim Bruce @ 2:41 PM 0 comments

 
A new twist on table sorting: the default sort order should vary with column type. If you request a sort on a numeric column, we should present the high numbers at the top (sort descending). On the other hand, a text column's default sort should be ascending so that it runs a-z.

Added a new sort code, sortdef_, which knows all this and does it for you.

Tidied up sort code generally. The colspec returned inside our IDynamicTable object reflects the actual state of the table in its most precise form, so you can request

.country.sortdef_gdp.

and obtain

.county.sortdesc_gdp.

in the returned object. This enables the client to reverse the sort order if requested.

T
- posted by Tim Bruce @ 12:19 PM 0 comments