Growth/Growth 2014/status
This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. |
Last update on: 2014-09-monthly
2012-04-monthly
Karyn Gladstone, Steven Walling, Maryana Pinchuk and Ryan Faulkner conducted the Necromancy experiment, emailing lapsed editors to encourage them to edit Wikipedia again. Work on the Template A/B testing project is wrapping up; a full report is expected in May. The E3 team will also be publishing details on each experiment on meta and the English Wikipedia. The technical specifications for each implementation will be posted on mediawiki.org. The team also began recruiting for its open positions; the first software engineer for the team will be joining mid May.
2012-05-monthly
The team started the development of the Timestamp Position Modification experimental feature, which was deployed then disabled due to a conflict between the ClickTracking feature and the MediaWiki API. Further testing and tuning continues, as well as analysis, redesign and development of the ClickTracking extension. We are gathering requirements for the next experiment on analyzing post-edit feedback, and we continue to hire software engineers.
2012-06-monthly
The team redeployed the Timestamp Position Modification experiment and it is now wrapped and in analysis. Designs and analytics work on the next experiment, post-edit feedback, were completed in preparation for a July deployment. Debug hooks were added to the clicktracking extension with the goal of improving QA for experiments. We wrote a clicktracking dashboard that intercepts event logging calls and displays them on-screen, shows which experiments are currently active, and to which bucket (if any) the current user has been assigned. Work is ongoing on a re-write of the clicktracking extension, which is taking shape as at Extension:E3_Experiments.
2012-07-monthly
The Timestamp Position Modification experiment was completed, and initial analysis shows that adding the timestamp on articles increases clicks on the History tab. Development started to deliver post-edit feedback messages; this experiment includes a proof-of-concept dry run of a new editor bucketing strategy for delivering experimental treatments that was deployed in advance of the full experiment. The team configured a test environment on Labs, to be used for UI and functional requirements validation by the team. The Wikimania conference was an opportunity to interact with editors from the English Wikipedia, and to define a new experiment related to cleanup templates.
2012-08-monthly
We deployed and ran the first iteration of post-edit feedback, testing whether various types of positive feedback after submission of an edit increase the productivity and retention of Wikipedia editors. (The results will be publicized soon.) We are currently working on the next iteration of post-edit feedback and on a new experiment which centers around the account creation process. We've also deployed click-tracking to the English Wikipedia community portal, account creation page, and the article edit form, and devised a tool for generating reports from the raw log data. Working with Asher Feldman, we've also architected an alternative data pipeline for event tracking, and begun its deployment.
2012-09-monthly
This month the E3 team announced the results of the first iteration of the post-edit feedback experiment, and worked on productization of the most successful confirmation message in a new extension, as well as through collaboration with the VisualEditor team. In addition, the team deployed the second iteration of experimental post-edit feedback, which lets new editors know when they reach important editing milestones early in their participation on Wikipedia. E3 also continued readying work on account creation user experience and the new event logging and usertagging analytics infrastructure to support feature experimentation, all of which are in alpha deployments to English Wikipedia.
2012-10-monthly
In October, the E3 team permanently deployed a confirmation message for all editors (Extension:PostEdit) to 16 Wikipedias, including six of the top 10 projects by size, and worked on associated maintenance of the feature. The team also deployed two iterations of tests for a new registration page (read more), including the beginnings of an API for client-side validation of the sign-up form. In support of current and future work, we deployed the beta EventLogging extension, a new architecture to replace the older ClickTracking extension. Last but not least, work started on redesigning the login and new experiments aimed at onboarding new Wikipedians.
2012-11-monthly
In November, the Editor Engagement Experiments team (E3) deployed the third and final A/B test of the new account creation page, including client-side validation. Results from basic data analysis of all three tests were published on Meta, and the project will now move to the productization stage. Extension:PostEdit was put in maintenance mode after being deployed to a further seven Wikipedias, including French and Portuguese. On the analytics side, E3 transitioned permanently to Extension:EventLogging for data collection purposes, and collaborated with the mobile team to track activity on Wikipedia's mobile beta. Last but not least, the team also deployed a small design improvement to the personal tools menu in MediaWiki core, in collaboration with the Language Engineering team.
2012-12-monthly
In December, the Editor Engagement Experiments team launched a new test aimed at Onboarding new Wikipedians. This interface delivers an optimized task list immediately after sign up, inviting those without an idea of how to get started to choose an article and try their hand at editing. The related GettingStarted extension was deployed mid-month and continued to evolve throughout the month, as early quantitative and qualitative research was conducted.
To go along with the launch of GettingStarted and other experimentation, EventLogging underwent heavy development, including the launch of a new Schema namespace on Meta for defining the data collected in a public, collaborative manner. We created production schemas for GettingStarted, account creation, mobile, and more. Ori Livneh also reworked the format, transmission, and cleanliness of data delivered to analysts and product managers, automatically generating database tables from these schemas for incoming events.
Late in the month, the team collaborated with fundraising to reach out to donors and readers as part of the annual fundraising campaign via email and a "Thank You" banner which ran at the end of the year. In addition to introducing millions of donors and readers to the Wikipedia editor community and inviting them to join, this campaign helped the team establish an experimental baseline for what a campaign to convert readers might look like.
In addition to the above launches, we continued development of the new account creation experience and Guided Tours by Matt Flaschen, which will be launched in January 2013. Active development was also begun by Ryan Faulkner and Dario Taraborelli on a user metrics API. The effort is threefold: to standardize user metrics in data analysis, to build infrastructure to efficiently compute metrics for a large set of users, and finally to expose those results via an API.
2013-01-monthly
In January, the Editor Engagement Experiments team ("E3") planned its goals for the quarter, which ends in March. We also made progress on the following projects which are included in that plan.
First up, we launched guided tours on the English Wikipedia, including a test tour to demonstrate the capabilities of the extension, and a tour associated with the "onboarding new Wikipedians" (aka GettingStarted) project. In addition to tours created by the team, the extension supports community-created tours. Note that unlike many other projects by the E3 team, guided tours are planned as a permanent addition to Wikipedia, with each tour implementation considered to be experimental. (For example: the "getting started" tour will be delivered via a split A/B test.)
While building guided tours, the team also A/B tested the Getting Started landing page and task list, measuring the effect it had on driving new contributions. Several rounds of analysis were completed and published on Meta (round 1, round 2), with the conclusion that the onboarding experience is leading to small but statistically significant increases in new English Wikipedians attempting to edit, as well as saving their first edit. In addition to measuring the effects of the guided tour associated with this project, immediate plans are to redesign the landing page and add additional task types, to entice more new contributors.
Work also continued on refining the reliability and precision of the data collected from EventLogging. In particular, we migrated EventLogging to a dedicated database, and began collecting server-side events in addition to client-side, to support work such as measuring account creations on desktop and mobile. January also saw the heavy use of the new User Metrics API, in order to complete cohort analysis of onboarding users and for metrics reported at the Board presentation on the Foundation's year-to-date progress. Development of the API continues, and a public announcement is expected for early March. Last but not least, a call was put out for a part-time Technical Writer to work on documenting both of these pieces of infrastructure.
2013-02-monthly
In February, the Editor Engagement Experiments team ("E3") continued working toward completing its goals for the quarter ending in March, which includes updates on the following projects.
After the intial launch of guided tours, Matt Flaschen and other team members worked on A/B testing the effectiveness of guided tours as part of the onboarding new Wikipedians experiences currently enabled on English Wikipedia. Results from these controlled tests are vital to understanding the impact of tours on editor engagement. In the meantime, the GuidedTour extension was enabled on Wikimedia Commons and six Wikipedias (including French, German, and Dutch), so that local administrators and volunteer developers could take advantage of the feature.
In addition to working on polishing and quantifying the effect of guided tours, significant progress was made on a new landing page for the onboarding project, with plans to launch early in March. The new Getting Started page will be expanded to include a wider variety task types offered to new editors. It will also be generated from a basic recommender system coupled with the GettingStarted extension, rather than relying on a bot.
Kirsten Menger-Anderson joined the team as Technical Writer mid-month. She began work with Ori Livneh, Dario Taraborelli, and others on documenting the EventLogging extension, with the goal of producing a comprehensive guide for end users of EventLogging, especially other Wikimedia Engineering teams in need of data. Future work by Kirsten will include similar documentation of the User Metrics data analysis API, which will be opened up for internal use in March.
2013-03-monthly
In March, the Editor Engagement Experiments team largely placed other projects – such as guided tours, EventLogging, and others – on hold to focus on two key initiatives: the "Getting Started" process for onboarding new Wikipedians, and on making the redesign of account creation and login a permanent, internationalized part of MediaWiki core.
For the Getting Started project, the team launched a new version on English Wikipedia, which included a new landing page with additional types of tasks suggested for brand new editors to try. The list of tasks is now generated by a basic recommender system built by Ori Livneh, which gathers, filters, and delivered a fresh list of tasks automatically for every editor. This new backend paves the way for releasing the "getting started" feature on other projects, after we've completed data analysis and testing to understand which kinds of tasks are ideal for first time editors. Additionally, Matt Flaschen collaborated with the Editor Engagement Features team to build notifications to welcome new editors and invite them to contribute via the Getting Started.
For the account creation and login work, S Page, Munaf Assaf, and the rest of the team rebuilt our design to work with MediaWiki core, and solicited reviews from outside the team. We currently plan to launch both interface redesigns on an opt-in basis in April, to have editors test the localization and other functional aspects of the forms via a URL parameter, before we enable them as default.
2013-04-monthly
In April, the Editor Engagement Experiments (E3) team focused first and foremost on its account creation and login redesigns in MediaWiki core. The first phase of the launch invited editors and readers on all Wikimedia projects to test the new forms on an opt-in basis, to identify bugs and localization issues across our many wikis. We expect to release these as the default forms in May, pending any final blockers.
For the team's Onboarding new Wikipedians project, we completed quantitative analysis of the latest version of the GettingStarted landing page, and began prototyping a new landing page and navigation system for usability testing prior to further development and launch, which is expected in early May as well.
On the analytics and infrastructure front, the team handed off the product roadmap for the User Metrics API to the Analytics team and colleagues in the Grantmaking and Programs department. Ori Livneh, in support of the data analysis needs on the team, began work supporting a Foundation instance of IPython Notebook.
Last but not least, the E3 team held its second Quarterly Review session, and began work planning its next high-level goals for the April–June quarter.
2013-05-monthly
In May, the Editor Engagement Experiments team (E3) launched its redesigns of account creation and login forms, after numerous bug fixes and working with local communities to customize the interface as needed. The initial rollout was to 30 of the largest Wikimedia projects in English and other languages, including Wikipedia, Wikimedia Commons, Wikidata, Wikivoyage, Wikispecies, Wikiquote, Wiktionary, Wikibooks, Wikiversity and Wikisource. Complete deployment as the default for all remaining projects was enabled early in June.
Also this month, the team launched and tested a major revamp of the "Getting Started" interface for onboarding new Wikipedians (in English). This included a redesigned landing page, a refactor of the backend to increase speed and stability, a new navigation toolbar on articles that new users were given as their first editing task, and a guided tour to help them complete their first edit. The results of A/B testing of this new version showed the largest increase in click-through rates for the landing page – up to 32%, a large increase over the 10-12% click-through rate of previous versions. Overall, it also showed a small but statistically significant increase in the rate of first time edits (+1.7%) by new English Wikipedians invited to participate in Getting Started.
Last but not least, the PostEdit extension (previously enabled on most of the top Wikimedia projects by size) was migrated to MediaWiki core. With this change, the post-edit confirmation message will be available on all projects, and will be more easily integrated in to VisualEditor.
2013-06-monthly
In June, the Editor Engagement Experiments (E3) team continued work on its experiments related to onboarding new Wikipedians, and launched several new extensions to Wikimedia projects.
First, the new Campaigns extension was added to all wikis. This analytics tool helps identify internal or external sources of new registrations, by adding a "campaign" name to the signup page URL. This month, E3 began running campaigns to learn about how many anonymous editors sign up on the top 10 Wikipedias, as well as how many sign up via the invitation to "Join Wikipedia" on the login page (see the list of active campaigns and analysis). Another piece of analytics infrastructure by the team is the new CoreEvents extension, which houses logging of MediaWiki core activity, like preference updates and page saves across all projects.
For the Getting Started project, the team conducted usability testing (see results and documentation) of new designs. E3 also refactored and refined the guided tours extension in June, including adding usability enhancements like new interface animations, support for community tours, and bug fixing. The team also planned and began work on an experiment to deliver guided tours to all first-time editors.
The team also assisted with A/B testing and research for VisualEditor before its July 1 launch date, assisting with experimental design, EventLogging instrumentation, and other work. After the VisualEditor launch, E3 started a week-long micro-survey of newly-registered users on English Wikipedia, to give us a first systematic look at the gender diversity of those creating accounts.
2013-07-monthly
In July, the Editor Engagement Experiments (E3) team made progress on a number of continuing projects. In terms of features, the team also completed work to integrate the onboarding new Wikipedians project with new infrastructural changes and feature releases.
For the GettingStarted, E3 collaborated with Platform engineering to ensure compatibility with the new "SUL2" cross-wiki authentication architecture. For the GuidedTour extension, the team completed a first release of support for guided tours of the VisualEditor interface, alongside tours of the legacy wikitext editor, and developed a plan to refactor the GuidedTour extension as well as its API. E3 also planned for its sixth A/B test of the GettingStarted workflow (see proposed specification and mockups). As an addition to the team's redesign of account creation and login (launched in May-June), we enhanced the design of the form for users who fulfill account creation requests for others.
E3 team member Matthew Flaschen also worked with two Google Summer of Code students on their projects. Richa Jain is working on the Annotator extension, which allows adding inline comments to a wiki page. Rahul Maliakkal is working on the Pronunciation Recording extension, for adding audio of pronunciations to Wiktionary.
On the experimental tools and data analysis front, E3 completed a significant rewrite of the Puppet configuration for EventLogging, our data collection pipeline, among other changes. For the MediaWiki-Vagrant portable desktop development environment, E3 added support for flexibly provisioning and unit testing extensions such as GettingStarted, GuidedTour, ParserFunctions, EventLogging, and others. Last but not least, the micro-survey of gender of new account registrations was enabled on German, French, Italian, and Polish Wikipedias, while data analysis on the English Wikipedia results began.
2013-08-monthly
In August, the Editor Engagement Experiments team (E3) primarily focused on development for its next and final A/B test of the Getting Started task suggestion system, a part of a project aimed at onboarding new Wikipedians. The team also worked on enhancements and bug fixes for the GuidedTour extension, such as adding the ability to customize default tour actions and better integration with VisualEditor.
During part of August, the majority of the E3 team was at Wikimania 2013 in Hong Kong, and delivered three talks, including on: guided tours, the team's new editor onboarding process, and product management at the Wikimedia Foundation.
2013-09-monthly
In September, the Growth team (formerly known as Editor Engagement Experiments, or E3), primarily worked on the onboarding new Wikipedians project. In particular, this included the creation and deployment of two new guided tours to teach any new user how to make their first edit, using wikitext or VisualEditor. The guided tours extension was also deployed to the following language editions of Wikipedia: Catalan, Hebrew, Hungarian, Malay, Spanish, Swedish, and Ukrainian.
Along with the renaming, the team held its third Quarterly Review (minutes are available), published its 2013–2014 product goals, and shared a new job opening for two additional software engineers.
In accordance with our 2013-14 goals, the Growth team began research into modeling newcomer retention on Wikipedia, anonymous editor acquisition, and article creation improvement.
2013-10-monthly
In October, the Growth team completed its sixth and final major A/B test of the GettingStarted and GuidedTour extensions for the onboarding new Wikipedians project. Data analysis results for this test were also completed and published, making way for rolling out the winning version on non-English Wikipedias in November. This month, the team also completed background research and early designs for its upcoming work on anonymous editor acquisition and Wikipedia article creation.
2013-11-monthly
In November, the Growth team primarily worked on refactoring the GuidedTour and GettingStarted extensions, including development of an API for the latter. This public API will be used by the Growth team, the Mobile team and others to deliver editing tasks to users across a variety of Wikipedia interfaces.
The team also spent significant time on the research and design preparations for its anonymous editor acquisition and Wikipedia article creation projects. This included participating in a community Request for Comment about a potential Draft namespace for articles, requirements gathering, and working on a Draft namespace patch.
Matthew Flaschen and Pau Giner attended the Wikimedia Diversity Conference and presented (along with Jared Zimmerman and Vibha Bamba) on how diversity related to the team's engineering and product work.
2013-12-monthly
In December, the Growth team spent time working on product development and research for upcoming Wikipedia article creation improvements. First and foremost, the team fulfilled a request from the English Wikipedia community to launch the new Draft namespace there. Pau Giner and others on the team simultaneously began design work on future improvements to drafts functionality (see blog post), including recruiting for usability testing sessions.
2014-01-monthly
In the first month of the year, the Growth team focused on two projects. First, we enhanced and refactored the GettingStarted extension, in part to support local configuration for different Wikipedias. The latest version of GettingStarted and GuidedTour will be released in English and 23 other languages in early February. Second, the team wrapped up several iterations of design and data analysis in support of upcoming work on Wikipedia article creation. We presented new designs for the Draft namespace, and completed a series of remote usability tests (see the results). We also finalized and published extensive quantitative analysis of trends in article creation across the largest Wikipedias. Last but not least, the Growth team welcomed its newest member in January, Software Engineer Sam Smith.
2014-02-monthly
In February, the Growth team first focused on releasing the new Wikipedia onboarding experience on additional projects. The GettingStarted extension was deployed to 30 Wikipedias, including all of the top 10 projects by number of page views. This marks the first time its task suggestions and guided tours were available outside English projects. The GuidedTour extension was also deployed to those projects (as a dependency of GettingStarted), as well as the Czech Wikipedia and se.wikimedia.org. Late in the month, the team also presented its work at its first Quarterly Review of the 2014 calendar year (see slides and minutes).
2014-03-monthly
In March, the Growth team primarily focused on bug fixing, design enhancements, and refactoring of the GettingStarted and GuidedTour extensions, which were recently launched on 30 Wikipedias. We updated icons and button styles, rewrote the interface copy, and refactored the interface to be more usable in non-English languages. We also began work on a significant refactor of the GuidedTour API, in order to support interactive tours that are non-linear. Non-linear tours will not depend on a page load to run, which will enable better support for tours in VisualEditor, among other things. Last but not least, we made progress on measuring the impact of GettingStarted across all wikis where it is deployed, with results for the first 30 days of editor activity expected in early April.
2014-04-monthly
In April, Growth switched gears to focus on a new experimental area: anonymous editor acquisition. The team prepared its first two experimental interface changes, aimed at asking anonymous editors to register accounts (expected to be launched in early May). The team also will be conducting basic research into the role anonymous editors play in Wikipedia − more at Research:Anonymous phenomena and Research:Anonymous editor acquisition.
2014-05-monthly
This month the Growth team launched its A/B test of two methods for asking anonymous editors to sign up on English, German, French, and Italian Wikipedias. Full analysis of the test results is expected in June, though preliminary data strongly suggests a positive impact on new registrations. We finished the mw.cookie module, assisted by Timo Tijhof. Matt and Aaron participated in the Zürich hackathon. Last but not least, Growth released two smaller enhancements to our data collection regarding article creation, including adding page identifiers to MediaWiki core deletion logs and tracking page restorations across all wikis.
2014-06-monthly
In June, the Growth team completed analysis of its first round of A/B testing of signup invitations for anonymous editors on English, French, German, and Italian Wikipedias. Based on these results, the team prepared a second version to be A/B tested.
Additionally, the team released a major refactor of the GuidedTour extension's API, as well as design enhancements like animations, a new CSS-based way of drawing guider elements, updated button styles, and more. The team also launched GuidedTours on three new Wikipedias: Arabic, Norwegian, and Bengali.
2014-07-monthly
In July, the Growth team completed its second round of A/B testing of signup invitations for anonymous editors on English Wikipedia, including data analysis. The team also built the first API and interface prototypes for task recommendations. This new system, first aimed at brand new editors, makes suggestions based on a user's previous edits.
2014-08-monthly
In August, the Growth team vetted CirrusSearch as back-end for personalized suggestions and prepared its first A/B test of the new task recommendations system. This test will deliver recommendations to a random sample of newly-registered users on 12 Wikipedias: English, French, German, Spanish, Italian, Hebrew, Persian, Russian, Ukrainian, Swedish, and Chinese. Several Growth team members also attended Wikimania 2014 in London. At Wikimania, the team shared presentations on its work and conducted usability tests of the recommendations system. Last but not least, design work began on the third major iteration of the team's anonymous editor acquisition project.
2014-09-monthly
In September, the Growth team shut down, with workflows shifting into the mainstream of other teams.