SwiftMedia/status

Last update on: 2012-05-18

2011-11-30

Ben Hartshorne has set up the initial development environment for Engineering to have a platform on which to test and continue development of the SwiftMedia extension. Ben and Mark Bergsma continue to do performance testing on Swift prior to using it in production. Aaron Schulz started to refactor the file backend code, a requirement to using SwiftMedia.

2011-12-20

Aaron Schulz merged the FileBackend branch into /trunk. Ben, Aaron and Ariel discussed sharding the swift containers.

2011-12-31

Aaron Schulz merged the FileBackend branch into /trunk, and Tim Starling started to review the code. Aaron, Ben Hartshorne, Ariel Glenn discussed sharding the Swift containers at the MediaWiki level, and Aaron started to implement it.

2012-01-03

FileBackend code has been merged, and has gone through preliminary review with Tim. Some fixes still being made in trunk. MediaWiki-level sharding of containers is currently being implemented by Aaron.

2012-01-05

Container sharding added FileBackend class. Aaron has a testwiki running of the swift test cluster and SwiftFilebackend.

2012-01-19

Aaron continues to make performance improvements and minor fixes to the Swift backend and FileOp classes.

2012-01-31

Aaron Schulz added container sharding to FileBackend, and continued to make performance improvements and fixes. The Swift back-end now passes all unit tests; the code is being reviewed and cleaned up. Some of Aaron's code for purging thumbnails will be backported to MediaWiki 1.18. Ben Hartshorne has prepared interim hardware for production deployment, which will begin the week of February 6.

2012-02-29

After more polish by Aaron Schulz and a February deployment of SwiftMedia and Swift, Swift is now serving 100% of thumbnails on Wikimedia Commons (engineering report).

2012-03-31

Swift is deployed for thumbnails. There are still some corrupted thumbnails in the Squid cache, but all known issues with new thumbnail corruption have been resolved. Work is underway to test and deploy Swift for original images, with work scheduled to complete in late May.

2012-04-monthly

Ben Hartshorne and Leslie Carr have been running a script to clean up corrupted thumbnails in Swift. Aaron Schulz has been fixing bugs, and is now focusing on adding concurrent file operation support to PHP cloudfiles (the PHP interface to Swift) and to MediaWiki's FileBackend code. This should help alleviate bug 34717..

2012-05-18

Aaron and Ben deployed a new version of the thumbnail handler to commons, test, test2, and mediawiki.org that uses our Swift FileBackend code, which should provide us useful production testing prior to using Swift FileBackend for handling originals. Cleanup of corrupted thumbnails is done now. Next up: Aaron plans to deploy a SiteStats fix today that should make uploads much faster and fix some timeout problems. Also, Ben and Aaron will roll out the FileBackend-based thumbnail handler to the rest of the wikis.