India hackathon (18–20 November, Mumbai, India) — Approximately 80 Indian participants came to this week-end hackathon, focusing on Wikimedia mobile, offline, and language issues. Participants made great strides in mobile. The internationalization effort also benefited, with new input methods for MediaWiki, readying Narayam for Wikimedia Incubator, a prototype onscreen keyboard built in Narayam, Wikimedia Mobile ready for translation, new UI prototypes for language selection, and other new features. Participants localized Kiwix into several Indic languages. Volunteer Development Coordinator Sumana Harihareswarasummarized the event on wikitech-l.
Brighton hackathon (19–20 November, Brighton, England) — Lewis Cawte and Tom Morris organized a general MediaWiki hackathon with approximately 10 participants. Attendees fixed bugs, reviewed outstanding patches from volunteers, enhanced OpenPlaques with the MediaWiki API, and discussed Wikinews and Wikimedia Commons.
Data Centers [?] — Eight new database servers for external storage were deployed in Ashburn and Tampa, to retire about 30 aging servers. The goal was to add capacity, improve performance, consume less power and provide cross data-center data recovery and redundancy (SDTPA & EQIAD). This gave us back one rack of server space, which is significant given the space constraint at SDTPA. Also in EQIAD, servers for bits.wikimedia.org and Fundraising reporting were migrated and are now in production. In Amsterdam, SSDs were added to the ESAMS Squid servers to improve read performance, and we upgraded the ESAMS core switch (csw2-esams). The KNAMS router was moved because we changed to a new hosting provider, though still within the same datacenter. In Tampa, 66 servers were retired, donated and shipped to various non-profits. More generally, our configuration management tool Puppet was upgraded and deployed to all of our lab and production servers; a Puppet dashboard was also implemented.
New MySQL Package — A new Debian package for mysqlatfacebook 5.1.53 was successfully built, using the fresh version of the facebook patchset. Initial production data on two hosts look promising.
Virtualization test cluster [?] — OpenStack Nova was upgraded from cactus to diablo. A GlusterFS filesystem was added on all compute nodes via puppet, to act as storage for the instances. A default sudo policy was also added for instances: project members now have sudo permissions, excluding global projects. Shared home directories are also available in a per project manner. 15 projects and 36 instances have been created and 46 people have been given Labs accounts so far. The GlusterFS installation and the recent Puppet master and client upgrades were implemented and tested in Labs before going into production.
Data Dumps [?] — While the dumps keep rolling along, we are talking with another organization interested in mirroring them. If you know someone with several terabytes of space who might be interested, please send them our way. All hosts had their kernel updated for security reasons, and new dump code was deployed as well. We rolled out a new experimental service this month of daily adds/changes dumps for all projects. No information about deleted/undeleted/moved pages from previous dumps is included, but it does include all new content since the run of the previous day. The first adds/changes run for the English language Wikipedia took less than 30 minutes to build.
Visual editor — Trevor Parscal fixed issues blocking the synchronization of structural edits to the user interface, refactored and cleaned up the code, and mapped out tasks and features to be supported. He also finished the document transaction functionality and made progress on an undo/redo system. Roan Kattouw added tests, rewrote some code to make the tests pass, and fixed a number of bugs and issues, notably in Internet Explorer. Inez Korczynski continued to work on content insertion, deletion and selection and fixed numerous bugs. Gabriel Wicke extended the PEG parser for robust larger-scale parsing. He converted the PEG parser into a combined wiki and HTML tokenizer that feeds to a HTML5 DOM tree builder. He implemented several wikitext features (lists, italics, bold) as token stream transformations. 139 of about 660 parser tests are now passing.
UploadWizard — Neil Kandalgaonkar and Ian Baker deployed a set of important improvements, including multi-file selection for browsers which support it, custom wikitext licenses, an improved licensing workflow, basic support for location data extraction, and more. Support for chunked uploading (which improves reliability of large file transfers) was temporarily backed out and is still being worked on.
Mobile Research [?] — Mani Pande and Parul Vora presented the findings from India and Brazil report to the mobile team. They are currently working on consolidating the report, videos, photographs and other media into a Wiki format. The cleaning of data from the mobile survey continues.
2011 Fundraiser — The annual fundraiser launched, with support for 76 new credit card currencies, including some which have been long-desired: Rupees, Russian Rubles, and Brazilian Reals. The fundraising engineering team added support to the DonationInterface extension for JCB credit card donations, BPay in Australia, 3 new real time banking options including iDeal, direct debit for 6 countries, manual bank transfers in more than 50 countries, and Webmoney, with more on the way. DonationInterface also benefited from enhancements to the RapidHtml form templating system. From an operations perspective, databases were migrated for increased capacity and stability; the data center failover capacity was also consolidated by adding aluminium.wikimedia.org as a mirror of grosley.wikimedia.org (different data centers, identical hosts), which provide a bulk of our miscellaneous fundraising services (including CiviCRM). We also added the ability to have translated 'thank you' emails to our ThankYou module for Drupal/CiviCRM. The ContributionReporting extension was enhanced to allow custom selection of fundraising years to display, but the feature was disabled after it caused a site outage due to cache stampeding. The FundraiserLandingPage extension was developed and deployed, making it easier to dynamically construct template calls for fundraiser landing pages depending on a potential donor's country. Last, the team fixed a number of new bugs and issues, surfaced by the increased usage of the donation pipeline.
Kiwix UX initiative [?] — Version 0.9 beta4 was released in November. Emmanuel Engelhart implemented a new filtering/sorting system in Kiwix's content manager. A new tool, kiwix-install, now makes it trivial to create new portable versions of Kiwix, with the ZIM file and full text search included. kiwix-install also made it possible to implement automated software and content updates. At the India hackathon, localisation for 3 new Indic languages was added, and the Kiwix team met with several organizations to discuss partnerships. Work has started on a Kiwix app for Android/ARM.
Continuous integration — Chad Horohoe and Antoine Musso created a Debian package for TestSwarm, so it can be installed in our common infrastructure with Jenkins. Antoine and Timo Tijhof wrote a script to fetch individual revisions of MediaWiki's code, in order to run tests against each one of them. PostgreSQL testing in Jenkins is planned for implementation in the coming weeks.
Git conversion — Chad Horohoe got a lot of help from the community on identifying unknown committers; user mapping is now complete. Gitorious was installed on a virtual machine, and a test git conversion of the MediaWiki code repository is now imminent. Meanwhile, there have also been discussions about e-mail aliases for LDAP users, to avoid disclosing private e-mail addresses. Chad and Brion Vibber worked on changes to the development workflow introduced by the move to git (e.g., when continuous integration tests get run).
Bug management — In November, Mark Hershberger and Sumana Harihareswara led themed bug triage sessions focusing on non-MySQL databases, MediaWiki 1.18 bugs, and UploadWizard. A session was also dedicated to reviewing patches in bugzilla; volunteer Rusty Burchfield wrote a tool to check if the patches could be applied to trunk, and only 50 were not obsolete due to bitrot. Mark watched for bugs and comments on local village pumps following the deployment of MediaWiki 1.18 to Wikimedia sites. He also continued to prioritize bugs and find developers to address those of highest priority.
Summer of Code 2011 — Sumana Harihareswara, Salvatore Ingala, Kevin Brown, and Yuvi Panda followed up on improvements and fixes, towards the goal of getting Ingala's, Brown's, and Panda's projects deployed on Wikimedia projects.
The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.