SQL/XML Dumps/Becoming a dumps co-maintainer/Access

Being a maintainer and developer for the dumps is a full stack job; we do capacity planning, coordinate orders of hardware, manage puppet configuration, write, package and deploy the code, monitor hardware and software issues, deal with upgrades of MediaWiki, PHP and Debian, and acommodate new requests for dumps. As such, there are a number of systems to which you will need specific kinds of access.

overview of sql/xml dumps components and their repos

Merging, testing and deploying code

To be a co-maintainer of the dumps with the ability to run all the tests you need to run, and merge all the patches in all the repos and deploy them, you will need:

Puppet

• +2 in the puppet repo ([1])

  (SRE) requires ops group in ldap and in puppet, or someone with it to merge for you (hnowlan, ariel)

• root on puppetmaster to deploy puppet patches

  (SRE) requires +2 in puppet, as these must be done together

• root on target hosts (dumpsdata, labstore1006,7, snapshot hosts) to disable, enable and run puppet

  needs new group dumps-roots in puppet with specific sudo privileges

• ability to run Puppet compiler jobs ([2])

  requires the wmf group in ldap, which all WMF developers should have

dumps code (python)

• +2 in python dumps repo ([3])

  requires membership in ops ldap group and in puppet, or someone who does (hnowlan, ariel)

  FIXME we should do something about this.

• deployment group on deployment hosts for scap

  requires membership in platform-engineering group in puppet

• admin access to deployment-prep for testing

  requires getting an existing admin ([4]) to add you via horizon

dumps code (C utils)

• +2 in mwbzutils repo ([5])

  requires membership in the operations-dumps-mwbzutils gerrit group

• root on apt repo host to add packages

  (SRE) requires membership in the ops ldap group and in puppet, or someone who can upload for you (hnowlan, ariel)

• root on target hosts (snapshot hosts) to install/update packages

  requires new group dumps-roots for the various commands on the snapshot hosts

dumps code (MediaWiki core, ActiveAbstract extension)

• +2 in core and ActiveAbstract repos ([6], [7])

  requires membership in the wmf ldap group, which all WMF engineers should have

• deployment group on deployment hosts for deployment of backports

  requires membership in the platform-engineering group in puppet

Server maintenance

Sometimes we need to upgrade to a new OS, add or replace servers, and so on. For these tasks you will need:

• root access to the cumin hosts, for re-imaging and other commands

  (SRE) requires membership in ops ldap group and in puppet, or someone who has it (hnowlan, ariel)

• root access to the management interface of the snapshot and dumpsdata hosts, for re-imaging, etc

  (SRE) requires access to pwstore or someone who can do this for you (hnowlan, ariel)

• root access to the snapshot and dumpsdata hosts, for rebooting or other actions

  requires new group dumps-roots in puppet with the appropriate commands

• access to icinga to downtime hosts/services or to ack alerts ([8])

  requires being in puppet/modules/secret/secrets/nagios/contacts.cfg or finding someone who is (hnowlan, ariel)

• access to grafana and other monitoring services, to check the state of the servers ([9])

  requires membership in the wmf ldap group, which all WMF engineers should have

• +2 in operations/dumps/scap repo for adding or removing targets from the dumps scap list ([10])

  requires membership in the operations-dumps-scap gerrit group

Watching for incoming issues

You'll need to be able to get warnings about issues as they come up, and notifications of patches to review.

• membership in the ops-dumps@wikimedia.org mail alias, for various statistics and exception reports

  requires someone with puppet merge and deploy privileges to add you

• get added automatically as a reviewer to relevant gerrit changes in MediaWiki

  add yourself with the right regexes here: mw:Developers/Maintainers

• become a member of (but not necessarily watch) the Phabricator dumps-generation project ([11])

  add yourself here: https://phabricator.wikimedia.org/project/members/1519/

• subscribe to and become a moderator of the xmldatadumps-l mailing list (mailarchive:xmldatadumps-l/)

  subscribe here: https://lists.wikimedia.org/postorius/lists/xmldatadumps-l.lists.wikimedia.org/ and get an existing admin to add you

Additional

It is sometimes useful to check what's happening with the vslow/dumps database servers. The tools of choice for that are dbtree ([12]), publically available, and tendril ([13]), requiring membership in the wmf ldap group, which all WMF engineers should have.