MediaWiki

Wikimedia Labs/Tool Labs/Roadmap en

< Wikimedia Labs | Tool Labs

Roadmap for the setup of Tool Labs and the migration from Toolserver

The German version is available here: Tool Labs/Roadmap de

Expectations of an infrastructure for volunteer software development

The Toolserver is an infrastructure that hosts software supporting Wikipedia and its sister projects. Our expectations of such an infrastructure are the following:

  • Any volunteer can request an account to develop tools to support, improve or analyze Wikimedia projects.
  • It makes sharing of code and knowledge as easy as possible.
  • The infrastructure is well maintained and available. Technical failures should be an exception.
  • The tools developed in such an infrastructure are running in an environment that is similar to the productive Wikimedia projects. The data of all Wikimedia projects and OpenStreetMap are easily queriable. By data, we mean the subset that is visible to normal, unprivileged registered users.
  • It offers user databases and shared storage for projects.
  • Tool maintainers can take part in the definition of features and software needed in such an infrastructures.
  • Ideally, it offers a staging process for software that can be used in production.
  • There are admins available, ideally in different timezones, that respond to the tool maintainers’ requests in a reasonable time. They should deal with maintainers with different amounts of technical knowledge and be open and welcoming to newcomers.

Crucial differences between the Toolserver and Labs/Tool Labs

  • Technical concept: The Toolserver is mainly used by individual developers. Though there is a concept of multi-maintainer projects, this is not considered to be the default. The ideas of sharing software and of offering a staging process are at the core of Labs’ concept though.
  • Responsibility: The Toolserver has been maintained by very few people for years. They were only partly paid and essential parts of the work have been done by a few volunteers. This has not only led to an immense work load for them but it has entailed a lack of documentation, too. This makes it more difficult to bring new sysadmins in to better share responsibilities. Labs/Tool Labs have been designed and built by paid WMF staff from the beginning. Volunteer admins are welcome but the responsibility for the project as a whole is assumed by an institution that has got more hands/heads to share the knowledge and more time for documentation. A single person leaving would thus not endanger the whole project.
  • Labs' dedication to OSI-approved licenses/free and open source software fits well into Wikimedia’s and Wikipedia’s concept of freeing knowledge. The Toolserver also hosts closed source software on both the level of infrastructure and tools.

General announcements of Wikimedia Germany

WMDE clarifies the following things:

  • The Toolserver is a community project. WMDE is in a coordinating role to support it, but the migration itself has to be done by the active community.
  • No tool maintainer willing to migrate their software shall be left alone. WMDE will try to assist them as much as possible. All tool maintainers who want support to migrate their tools are asked to tell us what exactly they need until the end of September 2013. (“Us” being either the Toolserver contact persons at WMDE or the WMF ops working on Tool Labs.) After this deadline, WMDE will not be able to take calls for support anymore.
  • The above mentioned commitment to free/open source software means though that no tool maintainer can claim the 1:1 transfer of their tools to Labs. It is possible that some tools will have to be ported to open source alternatives by the tool maintainers. WMDE does not accept responsibility for the preservation of all tools and WMDE will not be able to take care of proprietary software running on the Toolserver. According to WMF Legal, it is possible to negotiate exceptions: "If there are particularly important tools impacted by this rule, then on a case-by-case basis we may be willing to take a license to specific software. For example, if a university owns specific software, and are willing to license it to the Wikimedia Foundation and our volunteers (not just one specific volunteer), then we could discuss that. But the licenses would be very broad, and we would not be willing to negotiate them at great length or detail."
  • WMDE has run the Toolserver, WMDE is coordinating the transition process. We are not responsible for Tool Labs though - that is a WMF project.

Migration from toolserver - what WMDE offers to tool maintainers

  • Technical support for migration: Our software developer Johannes Kroll can be asked for help if tools need adaptations to run in Tool Labs. He will accept support requests until September 30th, 2013, he will help to estimate the amount of programming work needed. If desired, he can help to modify tools. A part of his working time is dedicated to this between October and December 2013. This is why he will prioritize, depending on the number of requests.
  • documentation of the migration process from Toolserver to Tool Labs

Status of Tool Labs (as of May 22nd, 2013)

Tool Labs consists of two projects inside Wikimedia Labs: "Tools" and "Tools-beta".

"Tools" is a stable production environment for tools. This project is built and maintained by three administrators, two of them being WMF staff and one a volunteer. More volunteers can be added, if they are ready to sign an NDA with WMF (like on the Toolserver). Root access is quite restricted here to keep the environment itself stable.

"Tools-Beta" is a more experimental environment for tools. This project is community-maintained and more flexible in its architecture so that tool maintainers can experiment with changes in the environment.

Overview of available features in Tool Labs

Topic Goal / Milestone
Database access Replicas of 6 out of 7 database clusters are available in Labs with private user data redacted. Cross-database join requests (WP/Commons) are possible with federated tables (links to tables on a different cluster). Federated database access needs to be requested for each database. (For no. 7 see below.)
Filesystem shared project storage and home directories (nfs)
Backup Backup for Tool Labs (for disaster recovery by ops only) is integrated with the general backup infrastructure. Users can access a backup of the last 3 hours, the 3 last days, and 2 last sundays.
Webservices Support for web tools is in place and functional.
Job-system There is a gridengine cluster deployed with support for punctual, scheduled and continuous jobs.
Support Two WMF system administrators and a volunteer are available for user support, plus WMF ops that will give extra support at the infrastructure level.
Bugtracker WMF’s Bugzilla has got components for Tool Labs in the "product" Wikimedia Labs. Migration of tickets from Jira isn’t impossible but someone would have to write a script for it.
Version control Git repositories can be requested in Gerrit for Labs. WMF doesn’t provide SVN hosting, but volunteers can create an SVN project in Labs.
Monitoring Icinga and Ganglia are in place: https://icinga.wikimedia.org/icinga/, http://ganglia.wikimedia.org/latest/
Access to instances http and sftp access are available. (For Tool Labs, there is a directly accessible proxy for the tools that does not need setup on a per-tool basis.)
Updates All instances are configured for unattended-upgrades to avoid security problems.
Logs / Statistics
"tools-beta" project Available features: public webserver for logs, CGI (perl, python) for helper tools, a variety of shells, php, java, libtcl, imagemagick, per-project optional custom MySQL databases (All databases will be accessible from all instances. Creation/modification/grants/etc. will be handled by sysadmins in the project that owns the database). There is a python version that is compatible with PyWikipediaBot.
"tools-beta" project Packages: Tool maintainers will be able to install software locally. Administrators have access to all packages in Ubuntu Precise and can pull and backport Raring packages when appropriate. When all else fails, a sysadmin can make a package and add it to WMF's own repo.
Use About two dozens of tools have migrated from Toolserver in the last six weeks or so (http://tools.wmflabs.org/). Feedback has been positive. The migration of simple tools is fast and easy.

Schedule

This road map clarifies

  • when Tool Labs is technically ready to replace the Toolserver,
  • which steps have to be undertaken to migrate.
Estimated Time of Completion Topic Goal / Milestone Responsibility
From 2013-05-24 on Database access Tool Labs offers database replication to the first testers. There are already volunteers for this.   Done Community
From 2013-05-24 on Database access See how many of the complicated queries could not be solved by federated tables nicely and deal with them separately together with WMF ops.   Done Answer: none found. WMF, community
From 2013-05-24 on Migration of tools RENDER starts to investigate its migration process.   Done WMDE
2013-05-24/25 Amsterdam Hackathon Workshop on migrating tools to Tool Labs, hands-on help   Done WMF, WMDE, community
2013-05-31 Migration of tools (depending on db replication) E-Mail is sent to tool maintainers asking for volunteers to move first and to document the process. WMDE, community
2013-06-12 Database access Replication of database cluster s7 (with Central Auth) is available.   Done WMF
Between 2013-06-15 and 2013-07-31 Migration of tools A small number of tool maintainers migrates their tools and serves as example (small and easy ones as well as a bigger and more complex one).   Done community (supported by WMDE and WMF ops)
2013-06-30 Logs Anonymization procedures are approved by WMF Legal.   Done WMF (ops and legal)
30.06.2013? (depending on OSM details, maybe RENDER, too) General status of Tool Labs Tool Labs has all necessary features to replace the Toolserver.   Done for 99% of use cases. All: WMF, WMDE, community
2013-09-30 Migration of tools Deadline for tool maintainers to ask for technical support for migration Community
2013-09-30 Migration of tools RENDER is running in Labs   Done WMDE
2013-09-30 Accounts New people interested in a toolserver account are sent to Tool Labs.   Done on TS wiki WMDE
1st half of October 2013 Clean-up Mail to all inactive accounts to download their data until Dec 31st 2013. Archiving script will be provided by TS admins.   Done WMDE
1st half of October 2013 Accounts All (single maintainer and MMP) accounts are set to expire at the beginning of January 2014. First renewal until June 1st 2014. Second renewal until June 30th 2014. WMDE
October 2013 to March 2014 Migration of tools Monthly reminders with information are sent to all tool maintainers to migrate or to backup their data somewhere else. WMDE
1st half of January 2014 Clean-up All ts account are set to expire.   Done Almost 300 users reactivated their accounts. WMDE
still open, details will be discussed in Amsterdam are being rediscussed OSM Technical requirements of OSM (Labs part) are defined and in place.   In progress Migration is ongoing. OSM, WMF
exact E.T.A. unknown, ca. end of Feb 2014 OSM Postgresql/postgis is running on dedicated hardware   Done WMF
E.T.A. unknown OSM Tileserver v.1 runs in its own project inside Wikimedia Labs   Done OSM@Toolserver Volunteers
Mid-March 2014 Infrastructure reloaded Wikimedia Labs have moved to equiad datacenter   Done WMF
Depends on Labs having moved Database access wikidatawiki and commonswiki are available on all clusters for federating WMF
E.T.A. unknown (Labs moved) Incoming e-mail Incoming E-Mail function for maintainers and tools is in place. (Outgoing e-mail is working.)   Done bugzilla:58796 WMF (ops and legal)
2014-06-30 Migration of tools All tools that shall move have moved. What is left on the Toolserver will be archived until August 31st, 2014 and then deleted before the final physical decommissioning. community (supported by WMF and WMDE)
2014-07-30 Decommissioning We have a plan for the Toolserver hardware that can still be used. WMDE and community
2014-07-30 Decommissioning We have a fixed date for the final (physical) decommissioning of the Toolserver. WMDE /WMF
2014-08-31 Decommissioning Deadline for requesting backups, data deletion ahead! WMDE
unknown / depending on OSM migration Decommissioning All data is purged from the Toolserver. WMDE, TS admins
unknown / depending on OSM migration Decommissioning Toolserver is decommissioned and donated to some other cool project WMDE, TS admins
2014-12-31 Buffer: Four extra months for unpredictable things. Everything is really, really done. All: WMF, WMDE, community
Retrieved from "https://www.mediawiki.org/w/index.php?title=Wikimedia_Labs/Tool_Labs/Roadmap_en&oldid=4939915"