Information about design, functionality and structure of Tool Labs for existing and new devs

Labs: a cloud-based infrastructure for developing and running tools run by Wikimedia Foundation
Tool Labs is a special part of Labs: It consists of two projects named "Tools" and "Toolsbeta". Inside Tool Labs you don't have to care / to know about Labs instances - the environment was made to resemble the toolserver (login server style).
The difference between the projects "Tools" and "Toolsbeta" is the following: "Tools" is a stable environment while "Toolsbeta" is a more flexible environment where you can experiment with changes in the environment itself for your tools.

Contact persons at WMF and WMDE

Wikimedia Foundation

Ryan Lane - Labs Operations Engineer, Timezone UTC-7
Marc A. Pelletier - Tool Labs Operations Engineer, UTC-4
Sumana Harihareswara - community liaison/documentation/workshops, Timezone UTC-4

Wikimedia Germany

Silke Meyer - project management @WMDE, coordination of the transition process, Timezone UTC+1
Johannes Kroll - software engineer, technical support for migrating projects, Timezone UTC+1
Lydia Pintscher - community liaison/communication, Timezone UTC+1

Mailing list and IRC

The central places to get help and to meet the crowd are

mail:labs-l, the mailing list. This is an excellent place for your questions. And subscribe to it too!
#wikimedia-labs ^connect, the IRC channel on the freenode network (more information).

Table of features and tools' dependencies needed for current tools

At the end of 2012, a list of features has been started. Feel free to add things! In the first part, we are collecting indispensable technical features Labs needs to offer what the toolserver has to offer now. Below, there is a list of features that are nice to have.

Wikimedia Labs/Toolserver features wanted

In addition, we are collecting technical details (like software and data dependencies) about tools on the toolserver to make Tool Labs fit for the existing tools. We need all tool developers' help to complete that list. Please edit it here:

Toolserver/List of Tools

List of available features

Marc Pelletier's to do list (regularly updated)

Schedule for the transition

A draft of the roadmap is available in English and in German. The bottomline: Tool Labs will be ready by the end of June 2013, tool maintainers will have one year's time to migrate their tools, and the toolserver will be decommissioned in the 2nd half of 2014.

WMF's milestones for the months to come can be found here: Wikimedia Engineering/2012-13 Goals#Milestones by quarter 2

List of important questions/FAQ

How will Toolserver tools work on Wikimedia Labs?

Tools project (stable environment)

Toolsbeta project (flexible/more experimental environment)

Do I have to set up my own virtual machine inside Labs? Do I have to know about puppet?

No. Have yourself added to the "Tools" or/and "Toolsbeta" project and you will find a login server similar to those on the toolserver. You don't have to care about what is underneath.

Do I have to migrate my software to Tool Labs?

You are all volunteers. We know that you are putting your free time into the development of tools to improve Wikimedia projects. We know that we are asking extra work by asking you to migrate your tools. Unfortunately, at some point in 2014, WMDE will discontinue the toolserver so that staying is no option. So we really encourage you to move over to Tool Labs.

When can I migrate my software to Labs?

Now. For many tools, database replication is a crucial prerequisite that is available now. Cluster s7 (with CentralAuth) is still missing, but upcoming in June 2013.

What do I have to think of when preparing migration?

Assuming that you want to copy files from the Toolserver to Tool Labs, please keep in mind that ssh/scp between the two currently works in one direction only (whyever). You can ssh from the Toolserver to Tool Labs but not the other way around.

What do I have to think of when finishing migration?

(This section is a draft, more explanations will follow!)

Wait a while to see if everything is okay. Then it's time to clean up on the toolserver.
Create a redirect to your tool's new URL and from all the old URLs in your account using .htaccess in your home. The redirects will be preserved after you leave.
(Find remaining deprecated links via access logs and correct them.)
Switch all your running processes off on the toolserver.
Do byebye, download your data, do burnbridges (on nightshade).

Where's the bug tracker?

WMF uses Bugzilla. Please use it for your issues!
There are the following categories in Bugzilla:
- product: "Wikimedia Labs"
  - component "bots" - deprecated
  - component "General" - Issues surrounding set up of labs projects.
  - component "deployment-prep" - Bugs relating to the configuration/management of the WMF deployment-prep project at http://beta.wmflabs.org. More info: wikitech:Nova Resource:Deployment-prep.
  - component "Infrastructure" - Bugs related to Wikimedia Labs infrastructure and not a specific Labs project.
  - component "tools" - issues related to tools running inside "tools" (and not "tools-beta") <- this is what you most likely use
  - component "wikistats" - http://wikistats.wmflabs.org/
If former Jira users need help / introductions, please don't hesitate to ask your questions on labs-l or on IRC (links above). Inform us about wishes for workshops! We are considering to offer workshops about software you are/will be working with in Labs that you didn't have on the toolserver. Tell us what you need!

Where is documentation?

What's a "service group"?

A 'service group' is a general Labs concept, known as a 'tool account' in the Tool Labs project. In toolserver jargon this is called multi-maintainer project, so a project group you can add other maintainers to.

What about file permissions? Who can see my code?

There are projects where users have root, so that all users in the project have full access to the whole project. This setup is not mandatory though: Tools can also use tool user IDs to control file permissions. On the tools project (which will be where toolserver tools migrate), you have full control over access permissions of your code and data. By default, only the tool maintainers have access (all the maintainers of a tool are in the tool's group).

Do stewards have a specific project on WMF Labs?

There is no plan to have distinct projects for different tool makers; but the tools are separated from each other. There is nothing that prevents you from sharing the maintenance of some tools between different stewards (in fact, it is recommended that you do so to ensure that there is always someone able to keep them up at need).

Can I delete a tool?

No, you can't do this yourself. The reason is that you or other members might accidentally delete precious stuff. You can delete the content of your directories. If you really want a tool / a service group to be deleted, please contact an admin.

If you are planning to try out Tool Labs but don't know yet if you are going to keep our tests as a later project, don't hesitate to create a tool (a service group) and to create a new one later where you put the stuff you want to keep.

Can I rename a tool?

No, sorry, this is not possible. You'd have to create a new one and put our code in there.

Can I have a subdomain for my web service?

Sorry, not yet. This is still in discussion at WMF. Currently, your web services are available under tools.wmflabs.org/<YOURTOOL>.

Do I explicitly have to specify the license of my tools?

Yes. If you think "this is just a draft, nothing ready" and you do not put a license into your code it's non-free software contradicting the idea of Tool Labs. So please add a license in the beginning!

How do I access the database replicas?

In your home directory you find your credentials for mariadb (in the file replica.my.cnf). You need to specify this file and the server you want to connect to. Some examples:

mysql --defaults-file=~/replica.my.cnf -h enwiki.labsdb # <- for English WP
mysql --defaults-file=~/replica.my.cnf -h dewiki.labsdb # <- for German language WP
mysql --defaults-file=~/replica.my.cnf -h wikidatawiki.labsdb # <- for Wikidata
mysql --defaults-file=~/replica.my.cnf -h commonswiki.labsdb # <- for Commons

Alternatively you can rename the credentials file from replica.my.cnf to .my.cnf and just run

mysql -h commonswiki.labsdb # <- for Commons

Why can't I access user preferences in the replicas?

The db replication gives access to everything that is visible for logged in users without special privileges. Others' user preferences are considered private information in Wikimedia Labs and are thus redacted from the replicas.

Anything I should know about Ubuntu Linux?

Not familiar with Ubuntu Linux? Below, you can find some good sources of information. You have to do with a Debian-based Linux distribution. Basically if there are different documents you'll probably want to look for the server (not the desktop) edition. The current stable release that has long term support is "Ubuntu Precise", a.k.a "Precise Pangolin", a.k.a. "12.04". This is what's running in Tool Labs. Once you know what a software/command line tool is called you can read the manual directly in Tool Labs by entering "man <name_of_software>".

Official Ubuntu Documentation: https://help.ubuntu.com/ Find groups or IRC channels in your language: http://loco.ubuntu.com/teams/

Help in German: http://wiki.ubuntuusers.de - very good explanations of software in German, command line as well as graphical tools. (In contrast, the forum.ubuntuusers.de contains relatively much bullshit.)

Please add more links to docs in different languages if you can really recommend them!

Will I be able to join user databases with wiki ones?

Yes. Databases created by users can be joined with the replica databases so long as they are created on the same DB.

Will the commons database be replicated to all clusters, like it is on the Toolserver?

It is not. However, there is a federated database link to commons and wikidata on every shard. There are some caveats in using them for good performance, but they are available.

Can I import my JIRA bug tracker information to Bugzilla? If so, how?

If you wish to migrate existing JIRA bugs to Bugzilla or use Bugzilla as a bug tracker for your tool for the first time, a Bugzilla component under the product "Tool Labs tools" needs to be created first. You can request this by following the instructions here.

Afterwards, you can use Merlijn van Deen's JIRA migration tool. To migrate your bugs:

In JIRA, under Administration, <your project>, People, add "Bugzilla Bug Exporter (valhallasw)" to "Administrators" and click "Update". Otherwise, the bot cannot close the bug.
File a bug in Bugzilla using this link.

Your bugs will be migrated for you! See this example for what the converted bug looks like.

If you are just interested in a backup of your project's JIRA bugs, but don't want to migrate to Bugzilla or another bug tracker (yet), JIRA provides XML dumps that are (almost) complete.

To download one, go to the JIRA search and type in for example "project = ". An autocompleted list should show up. Choose your project and click "Search". At the top right you should see a drop-down menu "Views". Choose "XML", save the file and look if it contains everything you expected.

You will need to download attachments separately as they are not included in the XML dump. You can find bugs that have attachments by looking for attachment elements.

Does the Tool Labs grid support qcronsub?

Not directly. Similar functionality is available with the -once option to jsub, but the common use case of "continually try to restart my tool if it failed" is better served with the support for continuous applications (-continuous, jstart).

Can I move a multimaintainer project to Tool Labs?

The short answer; all Tool Labs tools are "multimaintainer" as meant on toolserver (i.e.: shared project space, group membership)

My bot has many maintainers. How should we handle credentials?

Credentials the bot itself uses to edit/perform actions is subject to the target wiki's policies on sharing credentials and may need to be cleared with them.

Does Tool Labs support all the packages Toolserver does?

Yes, insofar as "If you need it and it is open source, we can get it for you." We do not, on the other hand, preemptively support every version of every dependency that has been enumerated; but do so on demand (partly to avoid installing obsolete packages that are no longer relevant).

Is there a backup of Toolserver SVN?

Yes, nosy made one. You can download the whole thing here (Internet Archive mirror).

How can chapters, communities and developers support the process

Now / Phase of preparations

You are running software on the toolserver? Please document what you need for it to work (link above)! Focus on functionalities rather than exact details, e.g. software versions.
Please talk to us if you are ready to become an early adopter to try Labs, e.g. once the database replication is up and running.

Know how and documentation

You are planning migration or did already accomplish it? Please help us to improve and add documentation. If you miss documentation, poke us. If you found out undocumented processes on your own tell us about them or write them down. If you don't know where to put documentation, don't hesitate to ask!
Would you like to give a talk or a workshop on a topic related to the Toolserver -> Labs transition at a Hackathon or online? Don't hesitate to propose it to us!

Hands-on support for others

You have migrated your own project successfully and feel like supporting somebody else? You are able to adapt software that needs modifications? Your help would be greatly appreciated! Please talk to us!

Financial support

Are you a chapter or some other project with funding, using the toolserver?

Please consider donating for the toolserver so it can run until we have got an alternative ready.
If you have paid developers in your projects, please consider paying them to migrate your project from the toolserver to Labs. You could even have them offer support to another / a related volunteer project if this is fine with the volunteers.

Wikimedia Labs/Tool Labs/Migration of Toolserver tools