Wikimedia Labs/status

Last update on: 2013-08-16

2010-10-01

Ryan Lane has been working on this since he started. He has been investigating OpenStack and other open source solutions. He's testing on a couple of machines. There's nothing production grade yet, but he's hoping to have something useful by the end of the year.

2010-11-01

Ryan Lane has investigated a few open source solutions, and has selected OpenStack. He's testing their newest beta release on a couple of machines. There's nothing production grade yet, but he's hoping to have something useful by the end of the year.

2010-12-01

We're in the process of getting hardware for this cluster, and we hope to have the system available by mid-January.

2011-01-01

New production hardware has arrived for implementation of this virtualization cluster. On the software and development side, Ryan Lane (and volunteers) have built virtual machine images for Nova testing, completed about 70% of the OpenStackManager MediaWiki extension, and integrated LDAP support into OpenStack Nova (read Ryan's article). In the office, Ryan also implemented a virtualized test server to reduce the need for copies of separate end-user operating environments (such as various configurations of Windows, IE, MacOS, etc.) for Features testing.

2011-02-01

Ryan Lane announced the release of the OpenStackManager extension for MediaWiki, which interacts with OpenStack, the EC2 API and LDAP to manage a virtual machine infrastructure (read more). However, there was a bit of a setback with some missing features in OpenStack testing, so we're waiting for the next OpenStack release before deploying this. We also have a little more hardware to configure as well.

2011-03-01

A new OpenStack has just been released, which contains the software features we need. This project was however also delayed due to the build out of the new data center. We expect to have the virtualization test cluster production ready in March.

2011-04-01

The deployment of the virtualization test cluster hardware (which was slightly delayed) is now ready for service. Ryan Lane released version 1.2 of his OpenStackManager extension and created detailed documentation on the setup. He will be finishing the deployment of the virtual test cluster in the first weeks of April.

2011-05-01

Production hardware is set up, and initial configuration of the software is done. We should have a basic environment demo-able in time for Berlin hack-a-thon. Ryan Lane gave a keynote at the OpenStack Developer's conference about this in late April (see the slides).

2011-06-01

MediaWiki is now installed, and most of the networking is configured. We are testing a couple new network configurations to avoid single points of failure in Openstack Nova's current design. We will begin installing instances in early June for basic infrastructure in the test/dev environment.

2011-07-01

Networking was set up, and instances can now be created and register with puppet. Work is now being done to move the puppet configuration into a git repository so that instances can fully build themselves. This project has been temporarily slowed down in favor of deploying HTTPS.

2011-08-01

This project was slowed down in favor of deploying HTTPS. Some work was done to move the puppet configuration into a public repository.

2011-08-31

Work has resumed on this project. The puppet configuration used in the production sites has been split into public and private repositories, and all sensitive information has been moved to the private repository. Gerrit has been configured, and the public puppet configuration will soon be moved into a public repository there. Labs LDAP and SVN LDAP are currently being merged, so that SVN users will more easily have access to Labs.

2011-09-30

All services except for DNS are up. The puppet repository has been released in a public repository. We've switched to using git/gerrit for our production puppet process. New instances have been tested building from scratch using puppet. We've started giving accounts out to some early testers of Labs via labsconsole.

2011-10-31

Wikimedia Labs has been launched. We are starting to install instances to duplicate our production environment. We are also bringing up some instances for non-MediaWiki related work. At the NOLA hackathon, we worked with a team from Canonical to give the juju project access to our environment. A new project from Wikimedia Deutschland, Wikidata, announced they'd be working in the Labs environment as well.

2011-11-30

15 projects and 36 instances have been created and 46 people have been given Labs accounts so far. OpenStack Nova was upgraded from cactus to diablo. A GlusterFS filesystem was added on all compute nodes via puppet, to act as storage for the instances. A default sudo policy was also added for instances: project members now have sudo permissions, excluding global projects. Shared home directories are also available in a per project manner. The GlusterFS installation and the recent Puppet master and client upgrades were implemented and tested in Labs before going into production.

2012-01-31

To keep up with project growth, the virtualization controller (virt1) has been converted into a compute node. Doing so let virt1 and virt4 join the instance storage, doubling the filesystem storage available. The additional compute node also allows Labs to grow by up to another 30 instances. An old Ruby gateway server (mobile2) was converted into the virtualization controller (virt0). A number of projects were added or moved to Labs, including incubator, ganglia, deployment-prep, globaleducation, a number of mobile projects, and a bunch more. Labs was very useful during the SF hackathon. A number of projects were created, implemented, and demoed using Labs during the hackathon. We also had a project created, implemented, tested, and deployed to production during the hackathon. We are still waiting for the gluster storage to arrive for volume storage; it should arrive early February. There are now 46 projects, 80 instances and 103 users.

2012-03-09

Added gluster storage cluster for project storage. Roughly 70TB of storage is available. Projects are quota'd at 300GB by default. Quota can be increased by request.

2012-09-26

The current infrastructure has been stabilized and most Q1 targets completed. Notably, OpenStackManager changes to show SSH fingerprints, along with DNS & Gluster support in Nova, are yet to be completed. However, we've also completed migrating to the Cisco hardware in pmtpa & adding MediaWiki support to Nova, which were scheduled for Q2.

2012-12-20

Ryan Lane and other staff are working on LDAP integration now, towards the goal of database replication that will enable more functionality for bots.

2013-04-05

Database replication for Wikimedia Labs work in progress (details)

2013-07-29

Database replication functional, Tool Labs in operation. See details.

2013-08-16

Tool Labs is now fully operational, and has dozens of bots and tools working in it full-time. The engineers behind Wikimedia Labs and Tool Labs presented two sessions at Wikimania in August, in addition to two workshops during the pre-conference DevCamp.