301: Moved permanently

It is a bad plan that admits of no modification.

Publilius Syrus (~100 BC), Maxims

We're now hosted completely on AWS, though we're not yet serving our page images directly from S3. Instead, our old hosting config is replicated almost byte-for-byte on EC2 and EBS (the elastic block store which enables storage to persist between virtual machine restarts).

In the long term there's definitely an argument to be made for moving all the page images onto S3, particularly when Amazon launches its Content Delivery Network on top of S3 later in the year. In the short term it's better to move one step at a time - doing it this way means we don't have to revise our import, logging, mail and hosting systems all at once. The mechanism are in place to migrate content over once we're ready.

So far (crossing all possible fingers) the new hosting is working fine. It's substantially cheaper than our previous arrangement and much more flexible. I've already upgraded the database machine, which involved

1. Starting a new machine instance


2. Shutting down the db


3. Disconnecting the drive from the previous instance


4. Connecting the drive to the new instance


5. Restarting the db


6. Telling the web servers to talk to the new machine


7. Terminating the original machine

All in all, 3 minutes' work and all done from the command line - no raising tickets or paying for unused hardware (running instances are billed by the hour). Awesome.

Also very cool are elastic IPs, which enable you to re-map your public IPs between running instances. This enables you to repeat the trick above with a web server. Will be trying that later in the month.

We're now running scheduled "snapshots" of our EBS drives. Every two hours the whole thing gets backed up to S3 - it only stores the deltas so the storage cost is low, and you can create a new EBS volume from any snapshot. Of course, all the snapshots are still within AWS so we still need an external backup.

NB: title is a geek joke. Our URLs haven't changed at all; they're just answered by different machines.

T

Labels: ec2, s3

- posted by Tim Bruce @ 2:29 PM 0 comments

 
EC2
Created a new AMI which will become our db server. Now running an instance of it with a newly-assigned Elastic IP address.

Next step is to create an EBS volume so the database's records persist between machine instances. Then modify the db config to use that volume and update the AMI.

S3 uploads continuing in the background. Yesterday I created a mechanism which will allow us to cache page details (in particular dimensions) in the db so they don't have to be gathered by each server independently. Also set up a queueing mechanism so new imports can be lined up for pre-caching.

T

Labels: ec2, s3

- posted by Tim Bruce @ 12:24 PM 0 comments

 
First title live
Our first public title is now on S3 - switched it over yesterday. Seems ok, but I still want to improve the speed of uncached pages a little.

Uploading the rest of the data tree to S3 in the background. Now that catapult knows when to leave existing files alone can re-run it as required to send any updates.

Logging is enabled for S3 requests.

DNS for our media alias is in place, but the service isn't yet using it - giving it time to propagate. Will enable it for the next release.

Current list of milestones for the transition:

1. Serve all page images from S3 (which will mean getting the import system to upload pages directly, also switching backups over to grabbing from S3).


2. Move index integration to EC2.


3. Move db to EC2.


4. Run a webserver instance on EC2, alongside our existing servers.


5. Switch mail handling to the EC2 server, routing through an external SMTP service.


6. Switch DNS to point to the EC2 webservers.

Next actions:

1. Speed up page generation for S3 content.


2. Package up catapult and integrate with import process.

T

Labels: s3

- posted by Tim Bruce @ 11:48 AM 0 comments

 
Release 6.8.9
New release is live - this adds:

1. Improved file handling (don't retrieve page images from S3 unless we really need them).


2. Eternal disk caching for S3 objects - objects no longer expire automatically.


3. Per-issue cache expiry on demand through a new admin toolbar

The Catapult tool now has support for MD5 generation but it doesn't use this yet - turns out that Amazon's eTag is explictly an MD5 of your content and this does correspond with the digests I'm calculating for the sample files. We can use this when checking for changed content in rsyncs and don't need to store our own MD5 as metadata for the object.

Checked ACLs for the bucket and test objects, those are fine - full_access for the owner and nothing more.

Finally to enable logging, then upload a live title and switch it over.

T

Labels: caching, release, s3

- posted by Tim Bruce @ 5:21 PM 0 comments

 
Caching nulls
Some of our pages don't have associated links or wordmaps (full-page images, for example).

Our new S3StorageManager used to return nulls to indicate that there was no such information - this has the drawback that while EHCache can store nulls as cache values, they can't be persisted to disk (as null is not Serializable). So after every restart the service would have to contact S3 again to establish that there was no wordmap for a particular page.

We're now returning empty byte arrays instead - these have the same meaning to the consumer of the information and can be cached between application shutdowns. Speed, speed...

T

Labels: caching, s3

- posted by Tim Bruce @ 12:31 PM 0 comments

 
Optimisations
The plan S3 for today:

1. Ensure we don't retrieve page images when we don't need to (for example, when preparing to calculate link rectangles for linkless pages).


2. Turn off expiry for the S3 disk cache.


3. Add ability to expire S3 cache on a per-issue basis.


4. Release.


5. Turn on S3 logging.


6. Get Catapult to store an MD5 of the object when uploading to S3 so we can "rsync" later on.


7. Re-check bucket settings, ACLs etc before uploading lots of content.


8. Set up DNS for media subdomain.

T

Labels: s3

- posted by Tim Bruce @ 11:34 AM 0 comments

 
Release 6.8.8
went live on Friday afternoon. This is the first public release of the Shibboleth link - there's a new option to log in "via institution". We're using the UK Federation's WAYF service for now; will implement a custom directory service when we support a client outside the federation.

For the moment it's time to shift focus onto the S3 migration. The site is now pulling all wordmaps, links, keywords, feature lists and page images from the appropriate service. There may still be a few locations which refer explicitly to local page images (cover thumbnails are a case in point) but there's enough there for testing.

Initial testing reveals that retrieval of uncached pages is too slow, owing to the round-trip between servers as we calculate the link rectangles. Once cached, the timing is fine.

T

Labels: release, s3, shibboleth

- posted by Tim Bruce @ 11:03 PM 0 comments

 
EHCache
is a fine, fine thing.

S3StorageManager is now caching the retrieved objects in a persistent disk cache. Next to bring LocalStorageManager up to speed, simplify the Highlighter and make a release.

The Shibboleth project is almost done. Our local federation gave the all clear last night, so we're now in their metadata. Did some initial testing with a real IdP today which worked initially but then broke - checked with the IdP that their metadata is up to date, as someone else had a similar problem for which stale metadata was the culprit. I must have tweaked something unfortunate. More testing on the way.

T

Labels: caching, s3, shibboleth

- posted by Tim Bruce @ 6:34 PM 0 comments

 
Persistent caches in Spring
The S3StorageManager is now picking up page images, word maps and links from S3 - woohoo! The next step is to enable caching so that it doesn't have to fetch a page image multiple times. The web servers will have plenty of free disk space so I'd like to use that to cache the objects from S3.

I had planned to use the AOP caching mechanism in Spring to take care of this automatically, storing objects in a persistent EHCache on the disk. Everything's configured and I can see the images getting pushed into the persistent cache, but they're never found again once you restart the application - so we'd get the benefits of caching while the app is running, but a site update or restart would mean we'd have to rebuild the cache from scratch.

A bit of poking around reveals that the CacheInterceptor is prepending the oject id of the target object to the cache key. I can see why that's a good idea (you probably do want to partition the method results cache contents by target most of the time), but it means that results cached from one object won't be used once that particular instance object's been replaced. In our case the target is a singleton anyway so we don't get any benefit.

I think the best solution might be for our S3StorageManager to manage its own cache explicitly. Off to check the EHCache documentation...

T

Labels: caching, s3, spring

- posted by Tim Bruce @ 5:29 PM 0 comments

 
Release 6.8.7
is live now - this adds provisional support for S3 hosting. I've swapped a disused title over to the new scheme and am uploading its data to S3. Testing tomorrow.

T

Labels: release, s3

- posted by Tim Bruce @ 1:09 AM 0 comments

 
Platform Selection
Titles can now be assigned to platforms (local or S3). Next to release this to the live service and move a test title across for larger-scale testing.

In other projects: a paper issue is on its way for test scanning, technical details are under review by the federation and the import queue is growing steadily...

T

Labels: projects, s3

- posted by Tim Bruce @ 6:40 PM 0 comments

 
Local S3 configuration
Spring's now autowiring our storage managers together - one bean each for the S3 and local managers, plus a composite manager which aggregates them into a single service. The S3 manager is configured in the per-machine settings so that we can keep the S3 details out of the codebase.

Introduced a new StoragePlatform enum which will become a field of the Magazine.

T

Labels: config, s3

- posted by Tim Bruce @ 1:12 PM 0 comments

 
Delivery from S3
Got S3 delivery working last night on my local test system. It's still at the proof-of-concept stage, but it seems fairly credible. Page images are delivered from S3 and come through about as fast as they do from our existing live servers.

There's a new storage abstraction layer which coordinates actions through an appropriate storage service; one for local and another for S3. Each title will be associated with one or the other service as we make the transition. The services are mainly stubs at the moment but will get fleshed out over the next couple of days.

The plan:

1. Pick up S3 connection parameters from our per-machine config file.


2. Add storage platform field to Magazine


3. Release


4. Upload all issues of a test title to S3 and switch platform


5. Test performance


6. Pick up wordmaps, links, keywords and page images through storage layer


7. Enable caching


8. Integrate index rebuilds

Unforseen benefit: my test server can now deliver anything that's available through the live site, as they're both serving the same content (well, it'll still use a local database, but the page images, links and wordmaps will all be common). No more staring at screens of placeholders while testing...

T

Labels: projects, s3

- posted by Tim Bruce @ 11:23 AM 0 comments

 
Catapult
Created a new tool (Catapult) which will upload a specified tree from our data hierarchy to S3. Seems to be working fine - next to create an abstraction layer in the web application so we can switch titles from local storage to S3.

Got the go-ahead from our federation, so we need to finish the live configuration for Shibboleth and share our metadata.

Also: imports. KW going up, need to do CH for publication later tonight.

T

Labels: projects, s3, shibboleth

- posted by Tim Bruce @ 6:58 PM 0 comments

 
Elision
Wow - was that really a week since the last post? Lots of things going on...

New site release is up (6.8.5). Features:

Improved login consistency - dropped the idea of a "theme" parameter being passed around, instead it's all now based on the title you're trying to access. If you came to the login page because you need to authenticate to see the page you requested, we base the login page theme on the theme of the title concerned. The title id is passed to the Forgotten Password page and back again to keep everything consistent.

All logins are now handled by the dedicated login processor. Previously the situation described above (you need to login to see what you requested) was handled by the page browser or issue browser - this led to inconsistencies where the preparation work done by our login processor wasn't taking place, so some options weren't getting offered to the user.

The year browser on the site's more accurate - we used to page your issue list into rigid chunks of 12 per page, then take you to the page containing the start of the requested year. Now we fine-tune the page start so it coincides exactly with the first issue from that year. This was a particular client request.

Some theme work for the same reason.

The load-balancer config's been updated to ensure that sessions are consistent across http and https requests.

Shibd still up and running in the test configuration. Waiting for confirmation from the federation before switching to a live config.

Also, imports and the regular monthly routines.

Going to clear remaining emails and dedicate today to getting S3 up and running for a test title.

T

Labels: projects, release, s3

- posted by Tim Bruce @ 10:41 AM 0 comments

 
S3 REST API in Java
Holding off on the Shibboleth integration tests until I can watch the logs throughout the day tomorrow.

In the meantime, experimenting with the Java library for S3's REST API. This seems to be more recently maintained than their SOAP alternative.

The sample application worked fine, storing and reading objects in a test bucket. I've compiled the integration code into a jar and created our own git repository of the current version.

Next to build a basic tool we can use from the command line to get some sample data into the service. Then to create an abstraction layer so we can pull the wordmaps, links and images (for highlighting/slicing) from S3 to the webservers. At the moment the webserver expects to find these on the local filesystem. We should be able to use Spring's caching framework to reduce the performance hit.

T

Labels: s3

- posted by Tim Bruce @ 12:31 AM 0 comments