Last update on: 2013-05-16
Chris, Sam, Antoine, Faidon, and Ryan met in San Francisco the week of May 7 to bootstrap work on this project. Current focus is getting media handling working smoothly.
As of May 15th:
- Apaches instances have been build 100% using puppet classes, the old one will be removed. All queries (thumbs/regular text/bits) hits the applications apaches, upload.beta.wmflabs.org pointing to the IP address shared by all wikis.
- MediaWiki logging is fine.
- Blocker: /home/wikipedia needs a decent place with lot of disk space to host MediaWiki checkouts, MediaWiki logs and syslogs bug 36868.
- Blocker: no syslog-server yet, since it conflicts with a base class which is always installed bug 36748.
- MediaWiki configuration files in progress of being merged from prod to labs.
Project is now a bit more on par with production status.
- A job runner has been setup, currently catching up with all the pending jobs. Apparently, that includes some video resizing for TimeMediaHandler.
- All code has been updated to a recent version and all databases have been upgraded.
- Uploading file should work again (as of May 17th)
Chris McMahon, Sam Reed, Antoine Musso, Faidon Liambotis, and Ryan Lane met in San Francisco the week of May 7 to bootstrap work on this project, kickstarting a process of aligning the configuration with our production cluster. Apache web server instances are now completely configured automatically using Puppet classes. A few key Wikimedia configuration files that were previously managed via private Subversion repository are now managed in a public Git repository. Much work remains to make this a stable testing environment, which will continue in June.
TimedMediaHandler has been setup though transcoding is not operational yet, since that would require a fully functional job queue. We discovered that the version of Ubuntu currently used in production (Lucid) won’t work with TimedMedia Handler. As a result, Antoine and Faidon updated the Puppet configurations for the Apache web servers to run on the next generation Ubuntu (Precise).
Administrative tools have been setup closely following the way it is done in production. As an example beta, use the exact same workflow to update the l10n cache. We will work on fetching l10n updates from translatewiki.
The primary focus of Beta cluster work in June was in service to TimedMediaHandler (TMH). TMH has been setup though transcoding is not operational yet, since that would require a fully functional job queue. The team discovered that the version of Ubuntu currently used in production (Lucid) won’t work with TimedMedia Handler. As a result, Antoine and Faidon updated the Puppet configurations for the Apache web servers to run on the next generation Ubuntu (Precise).
Administrative tools have been setup closely following the way it is done in production. For example, the Beta Cluster now uses the exact same workflow to update the l10n cache as we do in production. The team plans to further improve this by fetching l10n updates from translatewiki.
Beginning of July, the labs instances have been migrated to some new powerful hardware enhancing the performances by an order of magnitude. Some instances have been unfortunately corrupted in the process but thanks to our extensive use of Puppet, replacement have been pretty fast.
Antoine wrote an overview of the beta cluster, still need to be amended with sections about how to update code and debugging issues.
The MediaWiki code and extensions are now being updated on a regular basis. Petr Benan is starting implementing the IRC feed system for bots consumption. We received spammer attention, several counter measures have been applied such as the Captcha system enabled by Platonides and automatic blocking of known open proxies. The job queue system is being improved by Jan Gerber so it could fit in beta, that is a requisite for the Time Media Handler extension which would let us test video transcoding. Thumbnails are still not working correctly, a workaround is still being worked on.
All beta instances are now running out of the shared /data/project directory provided by the labs infrastructure instead of an NFS instance. Platonides has setup Captcha for user creation to help prevent spam, some well know IP have been banned. Jan Gabber is successfully using the infrastructure to work on Timed Media Handler, especially the job system that will process the video transcoding. Finally Ryan Kaldari is using the beta to setup E2 extensions.
The beta cluster infrastructure is now mostly in our configuration change engine (puppet) and start being used by third parties. The Features team and Jan Gerber are now taking advantage of the beta cluster to stage change for production. We have set up Captcha and IP blocking to reduce the amount of spam being generated on the beta wikis. An overview document has been started to help introduce new people to the beta cluster.
This past week has been focusing on cleaning out the cluster and working with ops to finish up the housework. All instances are now working on new hardware thanks to Andrew Boggot and all make use of the project storage path (/data/project) which was upgraded by Ryan Lane to use the latest GlusterFS release.
Most obsoletes and experimental instances have been removed.
The documentation has been expanded.
The MediaWiki core and extensions are now automatically updating. The beta cluster is from now always using the very latest version published under the master branch of each repositories.
The MediaWiki core and its extensions are now automatically updating, and the beta cluster is now always using the very latest version published under the master branch of each Git repository.
bits.beta.wmflabs.org is now fully managed by puppet. It serves MediaWiki and its extensions assets as well as geographical lookup of IP addresses http://bits.beta.wmflabs.org/geoiplookup
In September, QA Lead Chris McMahon announced that the Beta cluster is a fit test environment: code is routinely deployed there ahead of production, the test environment emulates the production environment closely, and we can easily and reliably manipulate aspects of the test environment (configuration, permissions, etc.) for testing purposes. Also, bits.beta.wmflabs.org is now fully managed by puppet. It serves MediaWiki and its extensions assets, as well as geographical lookup of IP addresses. Some work remains to be done (performance tuning, configuration) but the infrastructure is in place for software testing and browser test automation.
The MediaWiki configuration on the beta cluster has still a few remaining live hacks that prevent it from being upgraded smoothly. The final bits have been tracked down and will need a final sprint.
We are working on getting Zuul in place so Jenkins can talk to Gerrit - that's Antoine's goal. The goal for the NL hackathon is to work on CI in general. We hope that the NL hackathon will aid in speeding Beta cluster work. Filipin is working on getting CloudBees into a slave for WMF's Jenkins installation -- that's one of the NL hackathon goals.
Deployed AFTv5 to beta cluster and New Page Patrol is being maintained there as well. Still working on issues of ongoing maintenance.
Beta played a role in handling a recent issue with a defect that escaped to production. Beta remains the primary host for AFTv5 testing, including browser test automation
We deployed ArticleFeedbackv5 to the beta cluster, which is the primary host for AFTv5 testing, including browser test automation. New Page Patrol is being maintained there as well. We are still working on issues of ongoing maintenance, and this cluster played a role in catching a defect that recently escaped to production.
Arthur Richards is organizing a quick meeting to kick start the deployment of MobileFrontend on beta. Asked for support of PageTriage in test envs apropos of conversation with Ryan Kaldari https://bugzilla.wikimedia.org/show_bug.cgi?id=43203
The project to support MobileFrontend in Beta labs continues. We intend for Beta labs to become a test environment for the new git-deploy script from the Operations team: this should be helpful in ongoing maintenance of the environment
Kickoff held for Mobile Frontend support for beta labs, scope in place, project proceeding.
Support for Mobile Frontend on beta labs is underway, also proposed a novel use for beta to answer needs from the AFT project and E3.
The main use for the Beta Cluster in January was to test git-deploy. Zeljko Filipin continues to run regular tests there. Antoine Musso, Max Semenik, and Andrew Bogott are setting up MobileFrontend to run on Beta for testing purposes.
Work continued to add support for MobileFrontend on the Beta cluster, and a discussion to support a pre-release version of AFTv5 is underway.
MobileFrontend support nearly done. Support for other extensions under discussion, especially db updates
Automated support for database updates should be in place this week, which will greatly increase the value of beta cluster as a test environment for multiple extensions not well supported now.
Search being added to beta, MobileFrontend tweaks. Discussing new ways to use beta as a result of San Francisco gathering.
We are adding search to the beta cluster following Mobile Frontend tweaks. We are discussing new ways to use the beta cluster as a result of our San Francisco gathering in February.
Search being added to beta cluster in preparation for changing Lucene
Beta cluster db now being updated automatically! Added BZ issue to track moving automated tests to beta cluster from test2wiki. Search is up on beta but untested. https://bugzilla.wikimedia.org/show_bug.cgi?id=34250
"Phase 1" support on beta for Mobile is complete and Mobile is using the beta cluster for testing now. We added search to beta, including Mobile. Lucene instances have been set up to provide search suggestion and ... search capabilities, but it's a rough base which still needs to be improved. More automated tests are now targeting beta cluster, and targeting the test2wiki/production cluster is underway.
Jenkins is now upgrading the database schemas on an hourly basis and deploying changes to the MediaWiki configuration just after they have been merged. If you are curious, have a look at the Jenkins dashboard for the beta project.
Starting to point automated tests currently targeting test2wiki to beta labs also to shake out issues there and ultimately improve test coverage. Maintenance ongoing.
Beta as target for more automated browser tests in process. New utility on beta pays off: https://bugzilla.wikimedia.org/show_bug.cgi?id=47015
Pointed browser tests to beta labs, config of tests and beta to accommodate that is ongoing.
We started to point automated tests currently targeting test2wiki to Beta labs to shake out issues there and ultimately improve test coverage. This will help us with earlier detection of bugs introduced into master (such as bug 47015). Mark Bergsma and Antoine Musso refined the Varnish configuration for MobileFrontend, and further refined the configuration of the search functionality.
The Apaches and the bastion are now using a NFS server instead of GlusterFS. The pages are served much faster as a result (from 560ms down to 260ms).
Ariel Glenn and Antoine Musso started the work toward enabling SSL on the beta cluster. That will in turn let the Mobile and QA teams enhance their browser tests and will be very useful for the OAuth project.
Niklas Laxström enabled Universal Language Selector on the beta wiki ().
Max Semenik started setting up EventLogging on beta, he is supported by Ori Livneh.
HTTPS has been enabled on beta bug 36648, the certificate do not match the domains yet though.
Ariel Glenn has setup a redis instance on beta (bug 41530). Antoine fixed the jobrunner that was no more processing any jobs because it could not access to the Mediawiki files. Jobs are thus processing again, and from redis!