SQL/XML Dumps/Becoming a dumps co-maintainer/Access
Being a maintainer and developer for the dumps is a full stack job; we do capacity planning, coordinate orders of hardware, manage puppet configuration, write, package and deploy the code, monitor hardware and software issues, deal with upgrades of MediaWiki, PHP and Debian, and acommodate new requests for dumps. As such, there are a number of systems to which you will need specific kinds of access.
![](http://upload.wikimedia.org/wikipedia/commons/thumb/3/31/Sql-xml-dumps-repos-components.svg/300px-Sql-xml-dumps-repos-components.svg.png)
Merging, testing and deploying code
editTo be a co-maintainer of the dumps with the ability to run all the tests you need to run, and merge all the patches in all the repos and deploy them, you will need:
Puppet
edit- +2 in the puppet repo ([1])
- (SRE) requires ops group in ldap and in puppet, or someone with it to merge for you (hnowlan, ariel)
- root on puppetmaster to deploy puppet patches
- (SRE) requires +2 in puppet, as these must be done together
- root on target hosts (dumpsdata, labstore1006,7, snapshot hosts) to disable, enable and run puppet
- needs new group dumps-roots in puppet with specific sudo privileges
- ability to run Puppet compiler jobs ([2])
- requires the wmf group in ldap, which all WMF developers should have
dumps code (python)
edit- +2 in python dumps repo ([3])
- requires membership in ops ldap group and in puppet, or someone who does (hnowlan, ariel)
- FIXME we should do something about this.
- deployment group on deployment hosts for scap
- requires membership in platform-engineering group in puppet
- admin access to deployment-prep for testing
- requires getting an existing admin ([4]) to add you via horizon
dumps code (C utils)
edit- +2 in mwbzutils repo ([5])
- requires membership in the operations-dumps-mwbzutils gerrit group
- root on apt repo host to add packages
- (SRE) requires membership in the ops ldap group and in puppet, or someone who can upload for you (hnowlan, ariel)
- root on target hosts (snapshot hosts) to install/update packages
- requires new group dumps-roots for the various commands on the snapshot hosts
dumps code (MediaWiki core, ActiveAbstract extension)
editServer maintenance
editSometimes we need to upgrade to a new OS, add or replace servers, and so on. For these tasks you will need:
- root access to the cumin hosts, for re-imaging and other commands
- (SRE) requires membership in ops ldap group and in puppet, or someone who has it (hnowlan, ariel)
- root access to the management interface of the snapshot and dumpsdata hosts, for re-imaging, etc
- (SRE) requires access to pwstore or someone who can do this for you (hnowlan, ariel)
- root access to the snapshot and dumpsdata hosts, for rebooting or other actions
- requires new group dumps-roots in puppet with the appropriate commands
- access to icinga to downtime hosts/services or to ack alerts ([8])
- requires being in puppet/modules/secret/secrets/nagios/contacts.cfg or finding someone who is (hnowlan, ariel)
- access to grafana and other monitoring services, to check the state of the servers ([9])
- requires membership in the wmf ldap group, which all WMF engineers should have
- +2 in operations/dumps/scap repo for adding or removing targets from the dumps scap list ([10])
- requires membership in the operations-dumps-scap gerrit group
Watching for incoming issues
editYou'll need to be able to get warnings about issues as they come up, and notifications of patches to review.
- membership in the ops-dumps@wikimedia.org mail alias, for various statistics and exception reports
- requires someone with puppet merge and deploy privileges to add you
- get added automatically as a reviewer to relevant gerrit changes in MediaWiki
- add yourself with the right regexes here: mw:Developers/Maintainers
- become a member of (but not necessarily watch) the Phabricator dumps-generation project ([11])
- add yourself here: https://phabricator.wikimedia.org/project/members/1519/
- subscribe to and become a moderator of the xmldatadumps-l mailing list (mailarchive:xmldatadumps-l/)
- subscribe here: https://lists.wikimedia.org/postorius/lists/xmldatadumps-l.lists.wikimedia.org/ and get an existing admin to add you
Additional
editIt is sometimes useful to check what's happening with the vslow/dumps database servers. The tools of choice for that are dbtree ([12]), publically available, and tendril ([13]), requiring membership in the wmf ldap group, which all WMF engineers should have.