Roadmap for the setup of Tool Labs and the migration from ToolserverEdit
The German version is available here: Tool Labs/Roadmap de
Expectations of an infrastructure for volunteer software developmentEdit
The Toolserver is an infrastructure that hosts software supporting Wikipedia and its sister projects. Our expectations of such an infrastructure are the following:
- Any volunteer can request an account to develop tools to support, improve or analyze Wikimedia projects.
- It makes sharing of code and knowledge as easy as possible.
- The infrastructure is well maintained and available. Technical failures should be an exception.
- The tools developed in such an infrastructure are running in an environment that is similar to the productive Wikimedia projects. The data of all Wikimedia projects and OpenStreetMap are easily queriable. By data, we mean the subset that is visible to normal, unprivileged registered users.
- It offers user databases and shared storage for projects.
- Tool maintainers can take part in the definition of features and software needed in such an infrastructures.
- Ideally, it offers a staging process for software that can be used in production.
- There are admins available, ideally in different timezones, that respond to the tool maintainers’ requests in a reasonable time. They should deal with maintainers with different amounts of technical knowledge and be open and welcoming to newcomers.
Crucial differences between the Toolserver and Labs/Tool LabsEdit
- Technical concept: The Toolserver is mainly used by individual developers. Though there is a concept of multi-maintainer projects, this is not considered to be the default. The ideas of sharing software and of offering a staging process are at the core of Labs’ concept though.
- Responsibility: The Toolserver has been maintained by very few people for years. They were only partly paid and essential parts of the work have been done by a few volunteers. This has not only led to an immense work load for them but it has entailed a lack of documentation, too. This makes it more difficult to bring new sysadmins in to better share responsibilities. Labs/Tool Labs have been designed and built by paid WMF staff from the beginning. Volunteer admins are welcome but the responsibility for the project as a whole is assumed by an institution that has got more hands/heads to share the knowledge and more time for documentation. A single person leaving would thus not endanger the whole project.
- Labs' dedication to OSI-approved licenses/free and open source software fits well into Wikimedia’s and Wikipedia’s concept of freeing knowledge. The Toolserver also hosts closed source software on both the level of infrastructure and tools.
General announcements of Wikimedia GermanyEdit
WMDE clarifies the following things:
- The Toolserver is a community project. WMDE is in a coordinating role to support it, but the migration itself has to be done by the active community.
- No tool maintainer willing to migrate their software shall be left alone. WMDE will try to assist them as much as possible. All tool maintainers who want support to migrate their tools are asked to tell us what exactly they need until the end of September 2013. (“Us” being either the Toolserver contact persons at WMDE or the WMF ops working on Tool Labs.) After this deadline, WMDE will not be able to take calls for support anymore.
- The above mentioned commitment to free/open source software means though that no tool maintainer can claim the 1:1 transfer of their tools to Labs. It is possible that some tools will have to be ported to open source alternatives by the tool maintainers. WMDE does not accept responsibility for the preservation of all tools and WMDE will not be able to take care of proprietary software running on the Toolserver. According to WMF Legal, it is possible to negotiate exceptions: "If there are particularly important tools impacted by this rule, then on a case-by-case basis we may be willing to take a license to specific software. For example, if a university owns specific software, and are willing to license it to the Wikimedia Foundation and our volunteers (not just one specific volunteer), then we could discuss that. But the licenses would be very broad, and we would not be willing to negotiate them at great length or detail."
- WMDE has run the Toolserver, WMDE is coordinating the transition process. We are not responsible for Tool Labs though - that is a WMF project.
Migration from toolserver - what WMDE offers to tool maintainersEdit
- Technical support for migration: Our software developer Johannes Kroll can be asked for help if tools need adaptations to run in Tool Labs. He will accept support requests until September 30th, 2013, he will help to estimate the amount of programming work needed. If desired, he can help to modify tools. A part of his working time is dedicated to this between October and December 2013. This is why he will prioritize, depending on the number of requests.
- documentation of the migration process from Toolserver to Tool Labs
Status of Tool Labs (as of May 22nd, 2013)Edit
Tool Labs consists of two projects inside Wikimedia Labs: "Tools" and "Tools-beta".
"Tools" is a stable production environment for tools. This project is built and maintained by three administrators, two of them being WMF staff and one a volunteer. More volunteers can be added, if they are ready to sign an NDA with WMF (like on the Toolserver). Root access is quite restricted here to keep the environment itself stable.
"Tools-Beta" is a more experimental environment for tools. This project is community-maintained and more flexible in its architecture so that tool maintainers can experiment with changes in the environment.
Overview of available features in Tool LabsEdit
|Topic||Goal / Milestone|
|Database access||Replicas of 6 out of 7 database clusters are available in Labs with private user data redacted. Cross-database join requests (WP/Commons) are possible with federated tables (links to tables on a different cluster). Federated database access needs to be requested for each database. (For no. 7 see below.)|
|Filesystem||shared project storage and home directories (nfs)|
|Backup||Backup for Tool Labs (for disaster recovery by ops only) is integrated with the general backup infrastructure. Users can access a backup of the last 3 hours, the 3 last days, and 2 last sundays.|
|Webservices||Support for web tools is in place and functional.|
|Job-system||There is a gridengine cluster deployed with support for punctual, scheduled and continuous jobs.|
|Support||Two WMF system administrators and a volunteer are available for user support, plus WMF ops that will give extra support at the infrastructure level.|
|Bugtracker||WMF’s Bugzilla has got components for Tool Labs in the "product" Wikimedia Labs. Migration of tickets from Jira isn’t impossible but someone would have to write a script for it.|
|Version control||Git repositories can be requested in Gerrit for Labs. WMF doesn’t provide SVN hosting, but volunteers can create an SVN project in Labs.|
|Monitoring||Icinga and Ganglia are in place: https://icinga.wikimedia.org/icinga/, http://ganglia.wikimedia.org/latest/|
|Access to instances||http and sftp access are available. (For Tool Labs, there is a directly accessible proxy for the tools that does not need setup on a per-tool basis.)|
|Updates||All instances are configured for unattended-upgrades to avoid security problems.|
|Logs / Statistics|
|"tools-beta" project||Available features: public webserver for logs, CGI (perl, python) for helper tools, a variety of shells, php, java, libtcl, imagemagick, per-project optional custom MySQL databases (All databases will be accessible from all instances. Creation/modification/grants/etc. will be handled by sysadmins in the project that owns the database). There is a python version that is compatible with PyWikipediaBot.|
|"tools-beta" project||Packages: Tool maintainers will be able to install software locally. Administrators have access to all packages in Ubuntu Precise and can pull and backport Raring packages when appropriate. When all else fails, a sysadmin can make a package and add it to WMF's own repo.|
|Use||About two dozens of tools have migrated from Toolserver in the last six weeks or so (http://tools.wmflabs.org/). Feedback has been positive. The migration of simple tools is fast and easy.|
This road map clarifies
- when Tool Labs is technically ready to replace the Toolserver,
- which steps have to be undertaken to migrate.
|Estimated Time of Completion||Topic||Goal / Milestone||Responsibility|
|From 2013-05-24 on||Database access||Tool Labs offers database replication to the first testers. There are already volunteers for this. Done||Community|
|From 2013-05-24 on||Database access||See how many of the complicated queries could not be solved by federated tables nicely and deal with them separately together with WMF ops. Done Answer: none found.||WMF, community|
|From 2013-05-24 on||Migration of tools||RENDER starts to investigate its migration process. Done||WMDE|
|2013-05-24/25||Amsterdam Hackathon||Workshop on migrating tools to Tool Labs, hands-on help Done||WMF, WMDE, community|
|2013-05-31||Migration of tools (depending on db replication)||E-Mail is sent to tool maintainers asking for volunteers to move first and to document the process.||WMDE, community|
|2013-06-12||Database access||Replication of database cluster s7 (with Central Auth) is available. Done||WMF|
|Between 2013-06-15 and 2013-07-31||Migration of tools||A small number of tool maintainers migrates their tools and serves as example (small and easy ones as well as a bigger and more complex one). Done||community (supported by WMDE and WMF ops)|
|2013-06-30||Logs||Anonymization procedures are approved by WMF Legal. Done||WMF (ops and legal)|
|30.06.2013? (depending on OSM details, maybe RENDER, too)||General status of Tool Labs||Tool Labs has all necessary features to replace the Toolserver. Done for 99% of use cases.||All: WMF, WMDE, community|
|2013-09-30||Migration of tools||Deadline for tool maintainers to ask for technical support for migration||Community|
|2013-09-30||Migration of tools||RENDER is running in Labs Done||WMDE|
|2013-09-30||Accounts||New people interested in a toolserver account are sent to Tool Labs. Done on TS wiki||WMDE|
|1st half of October 2013||Clean-up||Mail to all inactive accounts to download their data until Dec 31st 2013. Archiving script will be provided by TS admins. Done||WMDE|
|1st half of October 2013||Accounts||All (single maintainer and MMP) accounts are set to expire at the beginning of January 2014. First renewal until June 1st 2014. Second renewal until June 30th 2014.||WMDE|
|October 2013 to March 2014||Migration of tools||Monthly reminders with information are sent to all tool maintainers to migrate or to backup their data somewhere else.||WMDE|
|1st half of January 2014||Clean-up||All ts account are set to expire. Done Almost 300 users reactivated their accounts.||WMDE|
|still open, details
||OSM||Technical requirements of OSM (Labs part) are defined and in place. In progress Migration is ongoing.||OSM, WMF|
|exact E.T.A. unknown, ca. end of Feb 2014||OSM||Postgresql/postgis is running on dedicated hardware Done||WMF|
|E.T.A. unknown||OSM||Tileserver v.1 runs in its own project inside Wikimedia Labs Done||OSM@Toolserver Volunteers|
|Mid-March 2014||Infrastructure reloaded||Wikimedia Labs have moved to equiad datacenter Done||WMF|
|Depends on Labs having moved||Database access||wikidatawiki and commonswiki are available on all clusters for federating||WMF|
|E.T.A. unknown (Labs moved)||Incoming e-mail||Incoming E-Mail function for maintainers and tools is in place. (Outgoing e-mail is working.) Done bugzilla:58796||WMF (ops and legal)|
|2014-06-30||Migration of tools||All tools that shall move have moved. What is left on the Toolserver will be archived until August 31st, 2014 and then deleted before the final physical decommissioning.||community (supported by WMF and WMDE)|
|2014-07-30||Decommissioning||We have a plan for the Toolserver hardware that can still be used.||WMDE and community|
|2014-07-30||Decommissioning||We have a fixed date for the final (physical) decommissioning of the Toolserver.||WMDE /WMF|
|2014-08-31||Decommissioning||Deadline for requesting backups, data deletion ahead!||WMDE|
|unknown / depending on OSM migration||Decommissioning||All data is purged from the Toolserver.||WMDE, TS admins|
|unknown / depending on OSM migration||Decommissioning||Toolserver is decommissioned and donated to some other cool project||WMDE, TS admins|
|2014-12-31||Buffer: Four extra months for unpredictable things.||Everything is really, really done.||All: WMF, WMDE, community|
- Overview of features needed in Labs/Tool Labs (community): Wikimedia Labs/Toolserver features needed in Tool Labs
- List of Tools (Silke and community, incomplete): Toolserver/List of Tools
- Migration Docs/FAQ (Silke, to be continued): Wikimedia Labs/Migration Of Toolserver Tools
- WMF Engineering goals: Wikimedia Engineering/2012-13 Goals#Wikimedia Labs
- Tool Labs Database Plan: wikitech:ToolLabsDatabasePlan
- List of tools hosted in Tool Labs: https://tools.wmflabs.org/