SwiftMedia

WMF engineering activity

SwiftMedia

Replacement of NFS with Swift on the Wikimedia cluster

Group:	Wikimedia Operations and Wikimedia Platform Engineering
Start:	2010-10-01
End:	2012-01-31
Management:	Ben Hartshorne
Team:	Aaron Schulz

Status

• [edit status] • [add new]
  2012-05-18: Aaron and Ben deployed a new version of the thumbnail handler to commons, test, test2, and mediawiki.org that uses our Swift FileBackend code, which should provide us useful production testing prior to using Swift FileBackend for handling originals. Cleanup of corrupted thumbnails is done now. Next up: Aaron plans to deploy a SiteStats fix today that should make uploads much faster and fix some timeout problems. Also, Ben and Aaron will roll out the FileBackend-based thumbnail handler to the rest of the wikis.

Rationale

The current Wikimedia storage architecture relies on a central NFS server for storage of media files. It's a single point-of-failure in our infrastructure, and scaling this out to multiple machines is difficult with the current MediaWiki design.

The goal of this project is to build a file store using Swift, and deploy Extension:SwiftMedia which is a new MediaWiki extension that interfaces with Swift file stores.

Roadmap

• Deploy SwiftMedia for purposes of storing thumbnails: scheduled for the week of February 6th
• FileBackend refactoring, and modifying SwiftMedia extension to use FileBackend
• Implement multi-file writing code to write to both NFS store and Swift store
• Deploy multi-file, primarily serving off of NFS store, while populating Swift store.
• Switch over to Swift store as primary store
• Decommission NFS store

Documents

• Swift page on wikitech - the full details of the Swift deployment.
• Status updates