Site performance and architecture

Wikimedia Foundation engineering activity

Site performance and architecture

Various initiatives geared toward improving site responsiveness, decreasing resource consumption, and/or improving site maintainability

Group:	Platform
Start:
End:
Team:	Asher Feldman, Ori Livneh, Aaron Schulz, Antoine Musso, Andrew Garrett
Lead:	Tim Starling
Status:	See updates

Status

[edit status] • [add new]

2013-04-monthly:

All job queues were migrated to JobQueueRedis off of the main DB clusters. Improvements were made to the category update queries to reduce lock exceptions that users often encountered when deleting files. This works via a new transaction callback hook added to the core database class, which can be used to resolve similar problems.

Rationale

Many small architectural changes and improvements are being done all of the time without a lot of fanfare. This is a general activity area where we communicate changes made along these lines.

Roadmap

April-June 2013

• JobQueue improvements
• Eqiad migration wrapup
  • Migrate fenari to tin.eqiad.wmnet
  • Migration to Ceph - still running sync scripts, possible split-brain issues with memcache
  • Migrate hume to terbium.eqiad.wmnet

Mysterious future

We plan to put the items below in a deployment infrastructure sprint sometime between July and December 2013:

• bug 46428 - mwscript.php/mctest.php does not know about memcache in both datacenters
• Database config cleanup -- multisite awareness in MediaWiki
• git-deploy/sartoris
• Better 500 error/PHP exception monitoring
• bug 47490 - resetUserTokens.php not usable on large wikis
• Improve file syncing with production on Apaches
  • Make updates atomic (e.g. symlink + directory move tricks)
  • Reconciling the use of timestamps on Javascript files (rsync vs ResourceLoader vs git)
• Kill deployment hacks with fire
  • live hacks that are still applied as of 2013-05-16

Documents

• Task management: Bugzilla
• Release management plan:
• Communications plan: