MediaWiki

Wikimedia Engineering/Report/2011/October

< Wikimedia Engineering | Report | 2011
(Redirected from Wikimedia engineering report/2011/October)
September 2011 Wikimedia engineering report, October 2011 November 2011

Major news in October include:

  • The New Orleans hackathon, which focused on Wikimedia's infrastructure;
  • Native HTTPS support on all Wikimedia sites;
  • Progress on the Visual editor project, with the first prototype expected in the coming months;
  • The deployment of the Translate extension to meta-wiki;
  • The deployment of MediaWiki 1.18 to all Wikimedia sites;
  • The completion of the first revision of the MediaWiki architecture document;
  • The ramp-up by the fundraising engineering team, to prepare for the upcoming annual fundraiser.

Hover your mouse over the green question marks ([?]) to see the description of a particular project.

Events

edit

Recent events

edit
  • New Orleans hackathon (14–16 October, New Orleans, USA) — About 30 MediaWiki developers and friends attended this coding event (organized by Ryan Lane, Sumana Harihareswara and Dana Isokawa) to work on Wikimedia's technical infrastructure. Work focused on the virtualization cluster / Wikimedia Labs, media storage (using Swift), Puppet, and continuous integration. The event was also used to discuss and prepare for the upcoming migration to git as our primary version control system.

Upcoming events

edit
  • San Francisco event (21-22 January 2012) — Erik Moeller and Sumana Harihareswara worked on planning for an outreach-focused developers' weekend. Staff and volunteer developers will help interested technologists learn to make awesome things with Wikimedia, by improving the Wikimedia platform, creating new functionality for Wikimedia users, or integrating Wikimedia content into their own products and services.

Personnel

edit

Job openings

edit

Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

One new position opened in October: Software Security Engineer, and an RfP was published: RFP/Mobile UI/UX Redesign.

The following positions are still open:

The following requests for proposals are still open:

Short news

edit

Operations

edit

Site infrastructure

edit
  • Data Centers [?] — Chris Johnson, Rob Halsell and Peter Youngmeister have racked and brought online new servers in Tampa to enhance the capacity of the Bits, MediaWiki and API application servers. Leslie Carr worked on partitioning with PartMan and documented the process. An external firm performed a security audit on our infrastructure on our behalf. The team also addressed several high priority break fixes, like the external data storage failures, and the analytics packet loss due to heavy load with udp2log processing. Peter Youngmeister upgraded all Apache servers to Ubuntu 10.04; Squid servers have also been upgraded, and the Linux kernels have been updated with the latest security patches. Ben Hartshorne continued to set up servers for external storage. Mark Bergsma and Asher Feldman deployed two new servers running a new and improved version of the Bits caching server in Amsterdam, to prepare for the deployment in Ashburn.
  • Media Storage [?] — During the New Orleans hackathon, developers agreed to create an abstraction layer (FileBackend) which will allow MediaWiki users to use SwiftMedia to store media files, or another filesystem of their choice. They also agreed to re-architect the FileRepo module in Mediawiki.
  • HTTPS — HTTPS support has been enabled on all Wikimedia sites; mobile support is still to come. At some point in the future, we'll be switching the log-in link to default to logging in via https.

Testing environment

edit
  • Virtualization test cluster [?] — Wikimedia Labs has been launched. We are starting to install instances to duplicate our production environment. We are also bringing up some instances for non-MediaWiki related work. At the NOLA hackathon, we worked with a team from Canonical to give the juju project access to our environment. A new project from Wikimedia Deutschland, Wikidata, announced they'd be working in the Labs environment as well.

Backups and data archives

edit
  • Data Dumps [?] — This month's dump for the English Wikipedia started later in the month (on October 11), since we were waiting for the production roll-out of MediaWiki 1.18. The run completed without issues, and all other servers are also using 1.18 without problems. We set up our first mirror site, hosted by the C3SL at Universidade Federal do Paraná in Brazil, whom we thank for their support. The mirror provides a copy of the last 5 successful dumps of each project and is updated as new dumps complete.

Features Engineering

edit

Editing tools

edit

Participation and editor retention

edit
  • Article feedback — Work has started on Version 5 of this feature, which will focus on reader engagement, for example by offering new feedback forms and free-text comment fields; a comment moderation system is also planned. This project is led by contractor Fabrice Florin, and will be implemented by an external contracting company, OmniTI.
  • LiquidThreads 3.0 — Andrew Garrett started working on a conversion script to facilitate the transition from the old schema to the new one.
  • Feedback Dashboard — The Feedback dashboard was deployed to the English Wikipedia, showing information provided by MoodBar. It is primarily read-only: it isn't possible to reply to comments. Andrew Garrett implemented the ability for administrators to hide comments which violate terms of use.

Multimedia Tools

edit
  • UploadWizard — Ian Baker implemented multi-file selection. Neil Kandalgaonkar did research on how licenses are used on Commons, added custom wikitext licenses with various safeguards to ensure this wouldn't result in more poorly licensed files, and implemented wording changes on the advice of Wikimedia general counsel. Ben Hartshorne implemented a simple way to add geocoding templates. Jeroen De Dauw also worked on this feature.

MediaWiki infrastructure

edit

Media Labs

edit

Mobile

edit
  • Mobile Research [?] — Mani Pande and Parul Vora conducted user experience research in the US in three cities (San Francisco, Chicago and Dallas) assisted by UX firm AnswerLab. They met with current and potential mobile readers, as well as editors. They are also finalizing the report on their field research in India and Brazil.
  • Android Wikipedia App [?] — We partnered with nitobi to create the first official Wikipedia app for Android devices. The app is based on the PhoneGap framework and should be added to the Android market soon. The team is still looking for developers.

Special projects

edit

Fundraising support

edit
  • 2011 Fundraiser — October was a very busy month for fundraising engineering, as the annual fundraiser is about to start. The biggest achievement was the rewrite and deployment of the credit card processing software, the DonationInterface extension. It now supports GlobalCollect, a new payment processor, in addition to the existing ones. Work is underway to add support for non-credit card payments. Another major feature of the extension is the ability to easily create new forms simply by inserting template tags into HTML markup; it also leverages MediaWiki's message translation system using translatewiki.net. Changes were made to the LandingCheck extension to determine whether to send users to donate.wikimedia.org (used by the Wikimedia Foundation) or wikimediafoundation.org (used by some Wikimedia chapters). Donators can now opt out from e-mails directly from the thank-you message they receive after their donation. Last, from an operations perspective, we increased the monitoring capacity of the servers in the payment cluster, as well as the redundancy of systems; we're currently working to implement automatic failover between our data centers.

Offline

edit
  • Kiwix UX initiative [?] — The last critical bugs in version 0.9 of Kiwix (especially on Windows) were fixed, and the release candidate cycle will soon begin. A Kiwix activity was released in alpha for Sugar, a learning platform for children. The first two prototypes of the Kiwix-plug have been built; the Kiwix-plug is a standalone wifi hotspot providing offline Wikipedia content, based on a plug computer.

Platform Engineering

edit

MediaWiki Core

edit
  • MediaWiki 1.18 — The deployment of MediaWiki 1.18 was successfully completed on all Wikimedia wikis. Developers are now fixing the remaining bugs before releasing 1.18 for third parties. The first beta is expected in early November.
  • Code review management — The focus on 1.18, as well as events and conferences, have allowed the backlog of unreviewed commits to increase in October (see chart). There are currently about 350 unreviewed revisions in trunk/phase3/, which means the deployment of MediaWiki 1.19 might not happen until January 2012.
  • Shell requests — Sam Reed continued to follow up on and fulfill shell requests.
  • Continuous integration — Chad Horohoe worked with the operations team to finalize the setup of the testing server, now online at https://integration.wikimedia.org. It is currently running Jenkins (PHPUnit), and TestSwarm should be added soon. Antoine Musso will be leading this project going forward.
  • Git conversion — Chad Horohoe started to prepare for the migration to git; he wrote a script to compare LDAP accounts to the information stored in Code review, and prepared for conversion tests from the current Subversion repository. An important step in the conversion is having names and email addresses for all committers, since neither Git nor Gerrit function well without them. As of this writing, 26 committers do not have any contact information, and 25 have a name but no e-mail address. Please help by filling in the names of committers you know.

Wikimedia analytics

edit
  • Wikimedia Report Card 2.0 — Erik Zachte discovered inconsistencies in the report card numbers, which were investigated and attributed to packet loss of up to 25%. This has now been fixed; further steps to prevent similar issues include adding monitoring to the process, and versioning the configuration files.

Technical Liaison; Developer Relations

edit

Future

edit

The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.

Retrieved from "https://www.mediawiki.org/w/index.php?title=Wikimedia_Engineering/Report/2011/October&oldid=6379254"