SQL/XML Dumps/Becoming a dumps co-maintainer/Access

Being a maintainer and developer for the dumps is a full stack job; we do capacity planning, coordinate orders of hardware, manage puppet configuration, write, package and deploy the code, monitor hardware and software issues, deal with upgrades of MediaWiki, PHP and Debian, and acommodate new requests for dumps. As such, there are a number of systems to which you will need specific kinds of access.

overview of sql/xml dumps components and their repos

Merging, testing and deploying codeEdit

To be a co-maintainer of the dumps with the ability to run all the tests you need to run, and merge all the patches in all the repos and deploy them, you will need:

PuppetEdit

  • +2 in the puppet repo ([1])
    (SRE) requires ops group in ldap and in puppet, or someone with it to merge for you (hnowlan, ariel)
  • root on puppetmaster to deploy puppet patches
    (SRE) requires +2 in puppet, as these must be done together
  • root on target hosts (dumpsdata, labstore1006,7, snapshot hosts) to disable, enable and run puppet
    needs new group dumps-roots in puppet with specific sudo privileges
  • ability to run Puppet compiler jobs ([2])
    requires the wmf group in ldap, which all WMF developers should have

dumps code (python)Edit

  • +2 in python dumps repo ([3])
    requires membership in ops ldap group and in puppet, or someone who does (hnowlan, ariel)
    FIXME we should do something about this.
  • deployment group on deployment hosts for scap
    requires membership in platform-engineering group in puppet
  • admin access to deployment-prep for testing
    requires getting an existing admin ([4]) to add you via horizon

dumps code (C utils)Edit

  • +2 in mwbzutils repo ([5])
    requires membership in the operations-dumps-mwbzutils gerrit group
  • root on apt repo host to add packages
    (SRE) requires membership in the ops ldap group and in puppet, or someone who can upload for you (hnowlan, ariel)
  • root on target hosts (snapshot hosts) to install/update packages
    requires new group dumps-roots for the various commands on the snapshot hosts

dumps code (MediaWiki core, ActiveAbstract extension)Edit

  • +2 in core and ActiveAbstract repos ([6], [7])
    requires membership in the wmf ldap group, which all WMF engineers should have
  • deployment group on deployment hosts for deployment of backports
    requires membership in the platform-engineering group in puppet

Server maintenanceEdit

Sometimes we need to upgrade to a new OS, add or replace servers, and so on. For these tasks you will need:

  • root access to the cumin hosts, for re-imaging and other commands
    (SRE) requires membership in ops ldap group and in puppet, or someone who has it (hnowlan, ariel)
  • root access to the management interface of the snapshot and dumpsdata hosts, for re-imaging, etc
    (SRE) requires access to pwstore or someone who can do this for you (hnowlan, ariel)
  • root access to the snapshot and dumpsdata hosts, for rebooting or other actions
    requires new group dumps-roots in puppet with the appropriate commands
  • access to icinga to downtime hosts/services or to ack alerts ([8])
    requires being in puppet/modules/secret/secrets/nagios/contacts.cfg or finding someone who is (hnowlan, ariel)
  • access to grafana and other monitoring services, to check the state of the servers ([9])
    requires membership in the wmf ldap group, which all WMF engineers should have
  • +2 in operations/dumps/scap repo for adding or removing targets from the dumps scap list ([10])
    requires membership in the operations-dumps-scap gerrit group

Watching for incoming issuesEdit

You'll need to be able to get warnings about issues as they come up, and notifications of patches to review.

AdditionalEdit

It is sometimes useful to check what's happening with the vslow/dumps database servers. The tools of choice for that are dbtree ([12]), publically available, and tendril ([13]), requiring membership in the wmf ldap group, which all WMF engineers should have.