Wikimedia Release Engineering Team/Monthly notable accomplishments

This page lists notable accomplishments for the month as we come up with them during our weekly team meetings.

January

 * Update dev images to split apache and php containers for local dev
 * Gerrit security bug discovery and deployed fix by Antoine
 * In sync with Gerrit upstream war (Java compiled code)
 * Target releases for apt packages in blubber deployed so wuvi can use npm

February

 * PipelineLib fully working on releases-jenkins.wikimedia.org
 * Rust introduction talk (not strictly RelEng business)
 * logspam-watch
 * Minimum hits consolidation feature
 * Error histograms, at-a-glance status indicators (emoji, it's emoji), improved UTF-8 handling and terminal resizing
 * Gearman plugin deployed. Merged bunch of pending changes + a fork from GoodData company which adds support for Pipeline jobs

March

 * PipelineLib fully working on releases-jenkins.wikimedia.org
 * Credentials added to pipelinelib
 * S&F contractors underway with production GitLab configuration
 * Terrible script for finding status of production errors on logstash dashboard
 * Ability to deploy phatality updates
 * scap apply-patches much improved

October

 * GitLab consultation

November

 * Gerrit security upgrade
 * Gerrit grafana dashboard
 * Created pipelinelib-experimental cloud project for working on pipelinelib
 * Scap 3.16.0 release (tagged, waiting on SRE now)
 * logspam-watch improvements
 * apparently scap apply-patches may possibly work in some circumstances
 * Upstream fix for shallow cloning in git: https://github.com/git/git/commit/fb3d1a083f776f02caa514cad8b232d8b974641f

December

 * Scap 3.16.0 released and deployed
 * Dropped scap plugins from mw-config
 * unconditional restart on deploy for opcache corruption deployed
 * https://doc-stage.wmcloud.org/, staging area for doc.wikimedia.org. Next prod then update related docs.
 * Scap source formatted with Black now
 * Runnable runbook blog

July

 * CI now supports REL1_35 branches (and ignores REL1_33).
 * Eliminate elasticsearch dependency from Phabricator search engine
 * Cassandra Docker image
 * Jenkins node Docker image cleanup & re-onlining after disk space recovers
 * Collection of disk space stats on Jenkins workers
 * Credentials and environment variables in PipelineLib
 * Blubber now correctly supports multi-stage artifact copies

August

 * Reduced the number of non-failure FAILURE messages in CI
 * After 9 months, Aphlict is finally back.
 * Scap version 3.15.0 released (in git, if not as .deb yet)

September

 * Scap 3.15.0 was deployed to all servers
 * We have trained another person to conduct the train
 * New phabricator metrics / stats in the project reports (currently deployed to cloud, prod coming soon)
 * image promotion in CI
 * Blog post about this: https://phabricator.wikimedia.org/phame/post/view/208/ci_now_updates_your_deployment-charts/
 * Tiny incremental improvements to logspam-watch (it now shows seconds)
 * GitLab consultation well underway
 * Released Quibble 0.0.45 https://doc.wikimedia.org/quibble/changelog.html
 * Local development mailing list and updates page

April

 * Docker images published on buster-based contint2001 (as part of general temporary switch-over from contint1001 to 2001 for buster migration)
 * Composer is now authenticated with github
 * Dropped basic PHP 7.1 testing from CI
 * Published Kubernetes migration tutorial
 * Phabricator milestone columns can now be moved on workboards
 * Phabricator workboards can be sorted by most recent activity.
 * Tech talk on PGP basics
 * "Cache of wmf-config/InitialiseSettings often 1 step behind" fixed! -

May

 * The release train branch cut is now an automatic job
 * Wikimedia Portals build and WDQS data release jobs moved to docker
 * The Continuous Integration instances on WMCS have been fully migrated off Jessie! T236576
 * Scap 1.14.0 released (by releng) and deployed (by serviceops)
 * Documentation for setting up a local dev environment for Phabricator: https://www.mediawiki.org/wiki/Phabricator/Local_Dev_Environment
 * CI server (contint) migrated to buster!

June

 * Scap plugins will move from mediawiki-config to scap git repository with the next release.
 * Deployment script added to deployment-charts for deploying to k8s
 * MediaWiki branch cuts are fully automated, at last!!!!
 * TMH job runner works in MediaWiki-Docker
 * Interactive logspam-watch
 * Gerrit 3.2.2

February

 * Production releases of Parsoid/PHP now also go through final pre-production tests
 * Scap release 1.13.0
 * Local development MediaWiki docker environment has shipped and been announced - https://lists.wikimedia.org/pipermail/wikitech-l/2020-February/093109.html / https://www.mediawiki.org/wiki/Docker

March

 * scap has its first integration test
 * MediaWiki tarball / Wikimedia production are now PHP 7.4-compatible.
 * All extension and skin repos are now being tested against PHP 7.4.
 * Analytics Refinery release job isolated into a Docker container.

December

 * PHP 7.4 testing was available in CI the first "business day" after 7.4.0 was released.
 * Revived "This week in logspam" email
 * Auto DBLists
 * PGP Key repo
 * Production config now has pre-merge diff reports, e.g.: https://integration.wikimedia.org/ci/job/operations-mw-config-php72-composer-diffConfig-docker/86/console

November

 * branch.py for cutting the branch for train
 * logspam-watch for tailing logfiles

October

 * Dev images are now automatically created as part of postmerge via the pipeline for:
 * Parsoid
 * Soon: RestBASE
 * (different from RESTbase? ;-))
 * Selenium documentation updated https://www.mediawiki.org/wiki/Selenium/Node.js
 * Quibble 0.0.36 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092658.html
 * Quibble 0.0.37 released https://lists.wikimedia.org/pipermail/wikitech-l/2019-October/092660.html
 * Quibble 0.0.38 & 0.0.39 released for mediawiki/tools/api-testing
 * Introducing Phatality - Streamlined error reporting from Kibana to Phabricator https://phabricator.wikimedia.org/phame/post/view/177/introducing_phatality/
 * HHVM removed from CI and MediaWiki.
 * Gerrit is on gerrit1001 now
 * … and so is most of the code review. ;-) :)
 * Unforked Jenkins Job Builder

September

 * Scap 3.12.1-1 released/deployed
 * Refactored Zuul layout to use per-branch pipelines
 * Lets you run arbitrary code against a working MediaWiki install
 * The phabricator "Report Error Code" form (https://phabricator.wikimedia.org/maniphest/task/edit/form/46/ ) has been updated with separate fields for the stack trace and error code/request id.
 * T232608 Delete selenium-daily-beta-EXTENSION Jenkins jobs that are broken more than 30 days
 * Write cached config to JSON as well as serialised PHP https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/533592/ (first step towards a saner config)
 * MediaWiki PHP support target modernised from 7.0+ to 7.2+ for 1.34 onwards. https://phabricator.wikimedia.org/T228342
 * Quibble 0.0.35 release
 * 1.34.0-wmf.24 branch cut was done /mostly/ with branch.py instead of make-wmf-branch.php (some small bugs remain to work out but it's very close)
 * Creating accounts was broken on beta cluster since 2019-09-08. It was fixed today (2019-09-25). https://phabricator.wikimedia.org/T232796
 * Phatality extension for Kibana deployed to production and used for reporting production errors into Phabricator.
 * Train blocker tasks created for 1.35.0-wmf.1—1.35.0-wmf.25
 * Dev images are now automatically created as part of postmerge via the pipeline for MediaWiki

August

 * Read only "gerrit-replica" active, handling 10% of all traffic (read from phab)
 * https://time.releng.team ¯\_(ツ)_/¯
 * Scap 3.12.0-1 in production

July

 * Migrated all generic CI jobs from PHP 7.0 to PHP 7.2 https://phabricator.wikimedia.org/T225457
 * Three new folks have been spun up on and have successfully run the Train, by end-of-month
 * it-phabricator plugin updated; fixes errors in All-Users repo in Gerrit
 * Completed first book club iteration: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Book_club/Continuous_Delivery
 * Unit vs Integration test split announcement: https://phabricator.wikimedia.org/phame/post/view/169/changes_and_improvements_to_phpunit_testing_in_mediawiki/
 * Gerrit 2.15.14 deployed
 * Contint1001 now storing docker images on separate partition
 * Blubber 0.8.0 deployed - https://lists.wikimedia.org/pipermail/wikitech-l/2019-July/092344.html
 * Deployment Pipeline docs published on Wikitech - https://wikitech.wikimedia.org/wiki/Deployment_pipeline

June

 * Speculative CI meta-architecture published within WMF for feedback (two versions)
 * Old image versions automatically removed from jenkins agents when /var/lib/docker space > 80%
 * scap 3.10.0 cut
 * Jenkins build timings reports: https://people.wikimedia.org/~dduvall/jenkins/
 * Helped Kask team sketch an outline of its architecture (https://www.mediawiki.org/wiki/Kask)
 * Fatal Monitor with marker lines for deployments: https://logstash.wikimedia.org/app/kibana#/dashboard/77cc3e90-aa27-11e7-9109-51bd3197f7a9?_g=

May

 * Repository-hosted CI/CD pipeline configurations now supported (.pipeline/config.yaml) - https://phabricator.wikimedia.org/T210267
 * Train notes published on branch cut
 * Codehealth pipeline beta - https://phabricator.wikimedia.org/phame/live/1/post/160/introducing_the_codehealth_pipeline_beta/
 * Some baseline local development images published

April

 * Phabricator vandalism rollback tool completed 🎉 (blog post? 😉)
 * Upgrade Zuul to 2.5.1-wmf6 (which unblocks the Gerrit upgrade to 2.16) - https://phabricator.wikimedia.org/T208426
 * Team offsite in Chicago

March

 * CI tooling future WG started, blogged
 * GerritBot comments on patches going through the pipeline (with fancy badges and the like)
 * Train deploy notes are now automatically generated on branch push
 * Scap 3.9.2-1 released in production
 * Phabricator upgrade: https://phabricator.wikimedia.org/phame/post/view/147/projects_forms_and_subtypes_oh_my/
 * Published the ISOSTWG results and recommendation on officewiki and announced: https://office.wikimedia.org/wiki/Internal_Support_for_Open_Source_Tools_Working_Group
 * swat tags now show up in the deployment schedule (via lua magic)
 * Blog post: https://phabricator.wikimedia.org/phame/post/view/152/help_my_ci_job_fails_with_exit_status_-11/
 * CI future WG report: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures_WG/Report
 * Blog post: https://phabricator.wikimedia.org/phame/post/view/155/quibble_hibernated_it_is_time_to_flourish/
 * Published a CLI tool to roll back vandalism in phabricator.

Feb

 * blubber uses blubberoid.wikimedia.org in the pipeline and pipeline is almost there for end-to-end functionality (can't yet deploy to production, but nearly can)
 * scap development back on gerrit -- new contributors
 * local-charts repo created
 * docker SIG announced/setup
 * Developer satisfaction survey results https://www.mediawiki.org/wiki/Developer_Satisfaction
 * Scap 3.9.0-1 released in production
 * Deployed wmf.18
 * Updated Phabricator to 2019-02-20 release, blog posted detailing some changes:  https://phabricator.wikimedia.org/phame/post/view/145/phab_phebruary/