Site performance and architecture/status

Last update on: 2013-04-monthly

2012-06-29

An initial investigation has begun on the possibility of upgrading from PHP 5.3 to PHP 5.4. Benchmarks are very promising, but a security enhancement we currently using with PHP 5.3 (Suhosin) is not yet available for PHP 5.4, so the team is debating whether to carry on without it, as well as estimating the performance penalty introduced by this patch. More improvements have been made to Ganglia and Graphite.

2012-06-monthly

An initial investigation has begun on the possibility of upgrading from PHP 5.3 to PHP 5.4. Benchmarks are very promising, but a security enhancement we are currently using with PHP 5.3 (Suhosin) is not yet available for PHP 5.4, so the team is debating whether to carry on without it, as well as estimating the performance penalty introduced by this patch. More improvements have been made to Ganglia and Graphite.

2012-07-20

Aaron and Andrew Garrett are working on job queue improvements. Tim is working on Apache configuration cleanup. Antoine is working on normalizing labs and production configurations

2012-07-monthly

Tim Starling investigated an LLVM PHP bytecode converter this month, which looked like a promising direction for performance optimization (slides here). The theoretical gain seems pretty significant, but actual performance he was able to observe was disappointing and we probably won't go in that direction. Asher Feldman has deployed an upgraded version of the parser cache server (db40) and the results have been impressive. Comparing 90th percentile and 99th percentile cache response times averaged over several days (July 3-5) for the parser cache server versus the last 8 hour for new improved parsercache shows 90th percentile response time dropping from 53.6ms to 7.17ms, and 99th percentile response time dropping from 185.3ms to 17.1ms. This is relevant to every page request from logged in and cookied logged out users so should have a meaningful impact on the user experience. Aaron Schulz and Andrew Garrett have been working on job queue improvements, Tim Starling on Apache configuration cleanup, and Antoine Musso on normalizing Labs and production configurations.

2012-08-monthly

In addition to the Lua work, Tim Starling did some investigation of parallel parsing, but that project may go on the backburner until after Parsoid goes into production. Tim Starling wrote a new Redis-based client for session handling. This will be important for the Virginia Datacenter Migration.

2012-10-monthly

Tim Starling committed changes to our implementation of libxml to use the PHP memory allocator, rather than using malloc (the C standard for allocating memory directly managed by the operating system). This will allow us to have per-page limits on the complexity of pages in a way that more closely mirrors their impact on our cluster.

2012-11-28

Support needed for more complex data structures (lists,sets) in memcached (with atomic updates) is awaiting more review and testing. The coding is essentially done (Change-Id: Ic27088488f8532f149cb4b36e156516f22880134).

2012-11-monthly

Various improvements to the job queue have been made to avoid CPU time wasted on duplicate jobs and redundant page cache purges. Changes have also been made to make it possible to edit heavily used templates without timeouts. Support needed for more complex data structures (lists, sets) in memcached (with atomic updates) is awaiting more review and testing. The coding is essentially done (changeset).

2012-12-monthly

After an assessment by Asher Feldman, Patrick Reilly and Tim Starling, the RDB database patch was canceled. Instead, in the short term, a separate vertically partitioned data cluster will be provided as a temporary storage until a horizontally scalable architecture can be finalized. Matthias Mullie is modifying the RDB-dependent ArticleFeedbackToolv5 to remove that dependency through an abstraction layer. When a sharded or horizontally scaled solution is provided, AFTv5's abstraction will be migrated.

An initial assessment of various non-MySQL alternatives for using Aaron Schulz's JobQueue core patch in 1.20 is being done for Echo. Because of the time it takes to exhaust the Echo queues, it is written to bypass the JobQueue through direct calls. Luke Welling is abstracting the JobQueue for Redis, ZeroMQ, and others.

2013-01-monthly

A patch to allow moving the DB job queue to another cluster is under review. An experimental redis-based job queue patch also exists in gerrit. Code was merged to support more complex data structures (lists, sets) in memcached (with atomic updates).

2013-02-monthly

A patch to allow moving the job queue to another DB cluster has been merged, and another patch to support an alternative Redis-based queue is in review in gerrit. Currently, job-related operations consume a significant portion of production database master wall time.

2013-03-monthly

JobQueueRedis merged, JobQueueAggregatorRedis merged and deployed. These improvements to the jobqueue should help site performance (see Tech Operations section of the monthly report).

2013-04-monthly

All job queues were migrated to JobQueueRedis off of the main DB clusters. Improvements were made to the category update queries to reduce lock exceptions that users often encountered when deleting files. This works via a new transaction callback hook added to the core database class, which can be used to resolve similar problems.