Article feedback/status

2010-09-01

We’re in the beginning phase of this project, figuring out requirements and generally determining the scope of our near-term and long-term efforts in this area. We are currently working on a pilot rating system which will be available as part of the Public Policy pilot program in late September.

2010-10-01

Phase 1, deployed last week in a pilot experiment for the Public Policy Initiative. The goal is to gather metrics and learn about how this feature should work. We plan to use this information to help develop requirements for the next phase in the first quarter of 2011.

2010-11-01

The first version of this feature was rolled out at the end of September, and we spent the month of October collecting and analyzing the feedback from the tool. We are currently designing the second version of this feature, and starting development later this month. We plan to roll out the next phase in sync with the Public Policy Initiative Phase 2 in January.

2010-12-01

In development. The initial version (phase 1) has been in production since the end of September on English Wikipedia. Phase 2, focusing on usability improvements, performance improvements and wider deployment, is slated for release in January 2011.

2011-01-01

In development. The initial version (phase 1) has been in production since the end of September on English Wikipedia. Phase 2, slated for release in January 2011, has been under development. Planning for phase 3 will begin soon.

2011-02-01

Phase 2 is currently in development. The requirements have validated against the current codebase and UI. This feature will be dark launched on prototype on February 4. We'll wait until after the 1.17 deployment before making this feature live due to Resource Loader dependencies.

2011-03-01

The deployment to our prototype has surfaced additional feature requirements that we've now addressed. Now that MediaWiki 1.17 has been successfully deployed, we can release the latest version of the Article feedback tool on the English Wikipedia, as part of our pilot experiment this quarter. Requirements for the next version (3.0) are being drafted.

2011-04-01

Phase 2 of the Article feedback feature

The second phase of this feature was released on the English Wikipedia in mid March. A major change in the interface is the ability for reviewers to specify the source of their knowledge, e.g. if they have an academic degree in a related field (see screenshot). Experiments to encourage user engagement are being performed as well. Dario Taraborelli also published an analysis of the first phase experiment. We're currently expanding the scope of the experiment to include several thousand articles, in order to get results that are more meaningful statistically.

2011-05-01

Phase 2: Trevor Parscal implemented the expiration of ratings, added error handling mechanisms and fixed IE bugs. Roan Kattouw reviewed and deployed the code to production.
Phase 3: Timo Tijhof and Trevor Parscal implemented the EmailCapture extension, a tool that allows unregistered users rating an article to leave their e-mail address if they want to be contacted later by the Community department. Weekly deployments were performed by Roan Kattouw. The team started to work on the dashboard, a summary page to surface general rating trends.

2011-06-01

Version 3 was deployed to the English Wikipedia on May 9 with new features, like the article feedback dashboard, a summary page showing general rating trends. The experiment was expanded to 100,000 articles and may be expanded further after analysis of the results. The next version of the Article Feedback feature is currently in development.

2011-07-01

Additional features were added in June, like a dashboard tracking articles receiving low ratings. Roan Kattouw started to implement the UDP back-end to provide clicktracking metrics to assess user engagement. The community provided feedback and bug reports, and the development team addressed the concerns raised, for example by implementing a user preference to hide the tool. Tooltips were also added to provide more information on the meaning of the star ratings. Dario Taraborelli continued to evaluate the data provided by the articles already showing the feature. The incremental roll-out to all articles on the English Wikipedia is planned to be completed by mid-July.

2011-08-01

Roan Kattouw completed the UDP logger (for clicktracking metrics) and deployed it to production. The Article feedback feature was incrementally rolled out to all articles on the English Wikipedia, and the Product research team continued to analyze its impact (read more).

2011-08-31

Development was completed in July; August was mostly devoted to data analysis. Dario Taraborelli analyzed the volume of edits, and couldn't yet find any statistically significant difference in edits before and after the activation of the feature on English Wikipedia articles. In order to clarify licensing and privacy policies regarding the data (and to facilitate its reuse by external tools & researchers), an explanation was published, stating that user feedback data (from Article feedback and MoodBar, for example) were considered public contributions just like any edit.

2011-09-30

Dario Taraborelli continued to analyze the data recorded. Howie Fung assessed proposals from vendors in response to the RfP to build an extended review system, that would notably integrate free-text comments to the Article feedback feature.

2011-10-03

Planning and design work begins on a new version 5 of the Article Feedback Tool. Fabrice Florin is engaged as product consultant to lead the development of AFT V5.

2011-10-31

Work has started on Version 5 of this feature, which will focus on reader engagement, for example by offering new feedback forms and free-text comment fields; a comment moderation system is also planned. This project is led by contractor Fabrice Florin, and will be implemented by an external contracting company, OmniTI.

2011-11-30

Fabrice Florin completed feature requirements, mockups and project plan for Version 5, and led development of first beta code by OmniTI. Dario Taraborelli started to work on the version 5 data model, metrics requirements and data analysis plan. He also worked with Fabrice and Howie Fung on guidelines and requirements for marking of AFT-generated edits and account tracking. A series of real-time dashboards for AFT data were uploaded to the Toolserver. Roan Kattouw reviewed the proposed database schema, and showed the OmniTI team how to deploy code to the prototype server.

2011-12-31

Aaron Halfaker, Oliver Keyes and Dario Taraborelli have finished gathering valuable data from the community about the usefulness of comments coming in from each of the three forms launched in December. A survey to get comments from readers about the effectiveness and attractiveness of each design was also introduced, and the team has been compiling the various sets of data to produce a report on the pros and cons of each form. Fabrice Florin is leading development on the next round of features, including a new feedback page, to be implemented by OmniTI, our development partner.

2012-01-31

Fabrice Florin led development on the next round of Article Feedback Tool v5 features, including a new feedback page, to be released in early February by OmniTI, our development partner. Aaron Halfaker, Oliver Keyes and Dario Taraborelli continued to collect valuable data from the community about the usefulness of comments coming in from each of the three forms launched in December. A survey to get comments from readers about the effectiveness and attractiveness of each design was also launched, and the team has been compiling the various sets of data to produce a report on the pros and cons of each forms in early February. The target date for an expanded feedback page is Feb. 15 for pre-deployment testing on en-labs, then wider deployment on Feb. 22.

2012-02-29

Fabrice Florin is working with our development partner OmniTi on version 5 of this tool, with the help of community liaison Oliver Keyes and analyst Dario Taraborelli. This month, the team created a new feedback page, with special features to be tested with oversighters and rollbackers in early March. Roan Kattouw has completed code review for compatibility with 1.19 and will deploy a new release on March 8, as well as a new working test environment in coming weeks. Final reports for phase 1 of this project will be published in early March.

2012-03-31

Fabrice Florin worked with OmniTi to develop new features for version 5 of the Article Feedback Tool (AFT5). This month, the team created new feedback links, new monitoring tools for editors and for oversighters, and started development on an abuse filter and a relevance filter. Brandon Harris and Heather Walls helped enhance the article feedback design, creating new icons for the monitoring tools. The AFT5 team also published its first report on phase 1 of AFT5, co-authored by Fabrice and Dario Taraborelli, with Howie Fung, Oliver Keyes and Aaron Halfaker. Roan Kattouw helped solve some tricky technical issues and deployed several new releases with the team this month. Current goals for this project are to complete feature development by the end of May, with full deployment in the summer.

2012-04-monthly

Fabrice Florin worked with OmniTi to develop a range of new features for version 5 of the Article Feedback Tool (AFT5). This month, the team deployed the first versions of the abuse filter and the relevance filter, as well as new monitoring tools for editors, to help surface useful suggestions and reduce the noise on the feedback page where posts are listed for each article. Pau Giner started designing a new look and feel for that feedback page, as well as a final version of the feedback form, with guidance from Brandon Harris. We also finalized a set of special tools for oversighters, as well as new feedback links. Dario Taraborelli, Oliver Keyes and Aaron Halfaker collected and analyzed data on how prominent feedback links impact both volume and quality of user feedback. Roan Kattouw continued to review our code and help deploy weekly releases, and Matthias Mullie contributed new code for this project. We expect to complete feature development by the end of May, with full deployment in the summer.

2012-05-monthly

Fabrice Florin worked with OmniTI and new WMF engineer Matthias Mullie to develop a range of new features for version 5 of the Article Feedback Tool (AFT5). This month, the team deployed a new look and feel for the article feedback page (based on a streamlined design by Pau Giner) as well as a central feedback page, where editors can monitor posts from all articles on Wikipedia. We also developed a final feedback form (scroll to bottom of page), which gradually engages users to contribute to the encyclopedia. Dario Taraborelli, Oliver Keyes and Aaron Halfaker collected and analyzed data on how posting feedback impacts both conversion and newcomer quality. Based on this analysis, we now project over 2 million feedback posts per month on the English Encyclopedia when the tool is widely deployed later this year (on par with the total number of edits per month). Our research suggests that posting feedback encourages a substantial number of users to productively edit articles on Wikipedia, which is expected to help reverse the recent decline in both new and existing editors. Roan Kattouw continued to review our code and deploy weekly releases, while training our team to deploy their own code over time. We are planning for a wider deployment by the end of June, with full deployment a couple months later.

2012-06-monthly

Fabrice Florin worked with new WMF engineer Matthias Mullie and OmniTI to develop a range of new features for version 5 of the Article Feedback Tool (AFT5). This month, the team completed primary feature development for this tool, including the article feedback page, the central feedback page, and the final feedback form (scroll to bottom of page). We started writing and publishing new documentation about this project, including this help page. Dario Taraborelli, Aaron Halfaker and Oliver Keyes published a full report that suggests that people who post feedback are more likely to edit articles afterwards. Roan Kattouw continued to review our code and trained our team to start deploying code on their own. We have started a wider deployment of AFT, which will gradually increase our coverage to 10% of the English encyclopedia by the end of July, with full deployment a couple months later.

2012-07-monthly

Fabrice Florin and Matthias Mullie led the deployment of Article Feedback on 10% of the English Encyclopedia, in collaboration with Pau Giner, Ryan Kaldari, Roan Kattouw, Oliver Keyes, Chris McMahon, Benny Situ, Heather Walls, Howie Fung and Terry Chay. This month, the team developed final features for this tool, including the article feedback page, the central feedback page, and the final feedback form (scroll to bottom of page). To guide users of this tool, we also published a new video tour, a walkthrough tutorial and various help pages. We have received a very positive response to article feedback from the Wikipedia community through a variety of channels, from talk pages to IRC chats and Wikimania presentations. Community members typically find the tool useful and well-thought out, and many editors have told us they have already made improvements to articles based on feedback from readers — which is exactly the behavior we were hoping to encourage. We have started our productization phase (more platforms, scalability, code re-factoring, localization, metrics, mobile). We are now aiming for a full release to 100% of English Wikipedia by October 2012 — with other wiki projects starting later this year.

2012-08-monthly

This month, we developed a range of new features for Article Feedback, which is now deployed on 10% of the English Encyclopedia. Improvements include the ability to view feedback from my watched pages, hide my posts, give feedback on help pages — as well as enable editors to clear all flags and administrators to protect articles to limit feedback on controversial pages. These and other features can be tested on this sample article feedback page or on the central feedback page (please report any bugs on Bugzilla). For more information about this tool, check our project overview. We are now in our productization phase (support for more platforms, scalability, code re-factoring, localization, metrics, mobile) and are aiming for a full release to 100% of English Wikipedia by the end of October 2012 — with other wiki projects starting later this year. This new tool was created in collaboration with the Wikipedia community and developed by Fabrice Florin, Matthias Mullie, Pau Giner, Ryan Kaldari, Roan Kattouw, Oliver Keyes, Chris McMahon, Benny Situ, Dario Taraborelli, Howie Fung and Terry Chay, in association with OmniTI.

2012-09-monthly

This month, we deployed more new features for Article Feedback, which is still being tested on 10% of the English Wikipedia. Improvements include iPad support, updating special abuse filters to automatically disallow swear words and common vandalism, as well as developing new automated filtering features to reduce the workload for editors, administrators and oversighters. These and other features can be tested on this sample article feedback page or on the central feedback page (please report any bugs on Bugzilla). We are also in the process of re-factoring our code and 'sharding' our database to make this extension more scalable, prototyping a mobile web interface, as well as collecting more data to track how many readers who post feedback end up becoming editors as a result. As we complete these final improvements and confirm that this tool is converting readers into editors as intended, we are preparing for a full release to 100% of English Wikipedia in coming weeks — with other wiki projects starting later this year. For more information about this tool, check our project overview. This new tool was created in collaboration with the Wikipedia community and developed by Fabrice Florin, Matthias Mullie, Pau Giner, Ryan Kaldari, Oliver Keyes, Chris McMahon, Benny Situ, Dario Taraborelli, Howie Fung and Terry Chay.

2012-10-monthly

This month, we developed a few final features for Article Feedback, which is being tested on 10% of the English Wikipedia. Though our lead developer was loaned out to the Wikivoyage project for most of the month, we deployed a couple improvements to reduce the moderation workload for editors, including: better abuse filters to automatically disallow swear words; new ways for moderators to filter the feedback page; and a check list to help users request oversight. These and other features can be tested on this sample article feedback page or on the central feedback page (please report any bugs on Bugzilla). We also started to collect new research data to track how moderators use the feedback page, as well as measure how many readers who post feedback become editors or registered users. Next month, we plan to make a few final improvements to address these findings, as well as complete code re-factoring to improve database performance. Once these tasks are done in coming weeks, we expect to release Article Feedback v5 to 100% of the English Wikipedia by the end of the year, and to other projects in 2013. For more information about this tool, check our project overview.

2012-11-monthly

This month, we continued to develop final features for Article Feedback, and researched how people are using this tool on the English Wikipedia. With the help of community members, we designed new features to reduce the editor workload, including improved moderation tools and a more prominent feedback link. These features will be developed next month, once we've completed code re-factoring to improve database performance. We also analyzed new research data to track how moderators use the feedback page, and measure how many readers who post feedback become editors or registered users. Next month, we will invite Wikipedians to evaluate the usefulness of feedback posts and the effectiveness of our new moderation tools. Once these tasks are done, we plan to release Article Feedback v5 to 100% of the English Wikipedia in early 2013. For more information about this tool, check our project overview.

2012-12-monthly

We made good progress on Article Feedback version 5 this month. We completed a research study on the English Wikipedia, confirming that many readers use this feature and a sizable number of them go on to register and become editors. Based on that research and editor suggestions, we started development on new features to reduce the editor workload through better filters and simpler moderation tools. We also continued to refactor our code, to support millions of comments on a dedicated database cluster to be deployed in coming months. Once this work is complete, we plan to release Article Feedback v5 to 100% of the English Wikipedia in March, and to other Wikimedia sites later this year. The German Wikipedia has already started a pilot to evaluate this tool, and a similar initiative is also under discussion on the French Wikipedia.

2013-01-monthly

In January, our team updated Article Feedback v5 and discussed its release with communities in the English, French and German Wikipedias. Developer Matthias Mullie completed a major code refactoring, which is now being reviewed. He also developed a final set of new features, such as simpler moderation tools and better filters, to be tested next month. Dario Taraborelli and Aaron Halfaker posted a feedback evaluation report, which suggests that about 39% of the feedback collected in their study can be used to improve articles (see also their other study results). Oliver Keyes responded to community questions in a request for comments about future deployments on the English Wikipedia, with a final decision expected next month. Fabrice Florin led product planning and discussed a possible deployment on the French Wikipedia and with the German Wikipedia, currently evaluating the tool in an ongoing pilot with a vote expected in May. Once our development is complete and communities reach their decisions for each project, we expect to release Article Feedback v5 on a range of Wikimedia sites in coming months.

2013-02-monthly

In February, our team completed feature development for Article Feedback v5 (AFT5) and prepared to release an updated version on English, French and German Wikipedias. Developer Matthias Mullie developed a final set of new features, including simpler moderation tools, better filters, a new feedback link, auto-archive and discussion on a talk page. Designer Pau Giner posted a usability study report about the effectiveness of the new moderation tools. Community liaison Oliver Keyes contributed to this request for comments, which concluded with a request to remove AFT4 and provide an opt-in version of AFT5 on the English Wikipedia (going forward, editors who wish to enable AFT5 for articles they watch can simply add the special Category:Article Feedback 5 on those pages). Product manager Fabrice Florin collaborated with members of the German Wikipedia (which is now testing the tool in its ongoing pilot, with a vote expected in May) and the French Wikipedia (which just voted to start its own six-month pilot). We plan to deploy AFT5's final version on these projects by the end of the month, as described in this 2013 release plan.

2013-03-monthly

In March, we released an updated version of Article Feedback v5 (AFT5) on the French and German Wikipedia, for evaluation by their communities. Developer Matthias Mullie completed final features for this release, including a new feedback link, auto-archive and a tool that lets you discuss useful feedback on article talk pages, based on designs from Pau Giner and suggestions from Oliver Keyes. Product manager Fabrice Florin worked with Denis Barthel and Sebastian Peisker on the German release and Benoît Evelin on the French release, and we are very grateful to them and many other community members for their invaluable contributions. A German community vote is expected in May on the German Wikipedia, and in October on the French Wikipedia, when they will decide whether or not to deploy the tool across their entire sites. Due to data caching issues, the tool was temporarily turned off on the English Wikipedia, where we expect to re-deploy it on an opt-in basis as soon as practical, as described on this talk page. After the English Wikipedia re-deployment in April, we plan to monitor community feedback on all three pilots before making any more updates, but other projects interested in the tool are invited to read this this 2013 release plan and contact us if they would like Article Feedback on their sites later this year.

2013-04-monthly

In April, we deployed the final release version of Article Feedback v5 on the English, French and German Wikipedias. Developer Matthias Mullie updated the back-end software in order to re-enable the tool on the English Wikipedia, and fixed a number of bugs reported on the German Wikipedia. Fabrice Florin worked with Pau Giner, Oliver Keyes and community members to simplify the feedback page, as well as finalize feedback links, auto-archive and opt-in features. Learn more in this project update. To enable feedback on articles you watch on the English Wikipedia, simply add the 'Article Feedback 5' category to these pages. For more tips on how to use this version, visit the testing page, and let us know what you think on the Article Feedback Talk page. We are now wrapping up development for this project, and will collect community suggestions for the next few months to prepare for upcoming votes on the French and German Wikipedias later this year.

2013-05-monthly

In May, we developed a few last features and bug fixes on the English, French and German Wikipedias. As requested by community members, Matthias Mullie developed a new opt-in feature that makes it easier to enable or disable feedback on a page, as well as new UI improvements to simplify the feedback page, based on designs by Pau Giner and specifications by Fabrice Florin. The team also released new feedback links and tested the auto-archive feature on prototype. For tips on how to use Article feedback, visit the testing page, and let us know what you think on this talk page. Feature development has now ended for this project, and we will determine our next steps based on the upcoming community vote on the German Wikipedia in coming weeks.

2013-06-monthly

In June, we deployed final features and bug fixes for the Article Feedback Tool (AFT5) on the English, French and German Wikipedias. Matthias Mullie released an opt-in feature to enable or disable feedback on a page, based on designs by Pau Giner and specifications by Fabrice Florin. In collaboration with Dario Taraborelli, Matthias also developed an updated set of metrics dashboards showing how the new moderation tools are being used: for example, about half of moderated feedback is marked as 'no action needed', while about a tenth is marked as 'useful' (these results are generally consistent across different languages). The team also supported a wider deployment of AFT5 on over 40,000 articles on the French Wikipedia, as well as a poll by the German community, which elected not to adopt the tool. Now that feature development has ended for this project, we plan to make AFT5 available to other wiki projects in coming weeks, as outlined in the release plan. For tips on how to use Article feedback, visit the testing page, and let us know what you think on this talk page.

2013-07-monthly

In July, we deployed a few last features and bug fixes for the Article Feedback Tool (AFT5) on the English and French Wikipedias. Matthias Mullie released the auto-archive feature, as well as this list of articles with feedback enabled on enwiki and on frwiki. At the request of the French Wikipedia community, he also developed new feedback notifications to let users know when feedback is marked as useful for a page they watch (or for a comment they posted). The team plans to make the AFT5 tool available to other wiki projects interested in testing this tool, provided that no new development is required to support their needs, as outlined in the release plan.

2013-08-monthly

In August, we made a few feature tweaks and bug fixes for the Article Feedback Tool (AFT5) on the English and French Wikipedias. Matthias Mullie released a few patches to improve the opt-in/opt-out tool, and tested the new feedback notifications to let users know when feedback is marked as useful for a page they watch (or for a comment they posted). We also presented our work on AFT5 at Wikimania 2013, with designer Pau Giner and our French and German champions Benoît Evellin and Denis Barthel, in this session (see slides). The team plans to make the AFT5 tool available to other wiki projects interested in testing this tool later this year, as outlined in the release plan.

2013-10-monthly

There was no activity on Article Feedback in October, as we await the results of the French pilot to determine our next steps for this project.