Article feedback/FAQ

What is the "Article Feedback Tool"?

The Article Feedback Tool (AFT) is an experimental feature that allows any readers of an article (whether they're editors or not) to quickly and easily assess the sourcing, completeness, neutrality, and readability of a Wikipedia article on a five-point scale. It's a way to increase reader engagement by getting feedback from readers on how they view the article, and it gives editors an easy way to see where an article needs improvement. In general, the pilot of this tool also reflects a shift in the Wikimedia Foundation's development processes towards more systematic experimentation and trials with new technology. We believe small experiments like this can be very useful in helping Wikimedia to innovate and learn.

How do I use the tool?

Just click on the star you want to assign each article and press the submit button. You can see average ratings for the same article by other users by clicking the View page ratings link.

How do I disable the tool?

To prevent the Article Feedback Tool from appearing in any page, you must be logged in as an editor to a site such as Wikipedia that uses the tool. Go to the "Appearances" tab of "My preferences", and check the "Don't show the Article feedback widget on pages" option in the "Advanced options" panel.

The Category:Article Feedback Blacklist can be added to specific pages and/or templates to avoid the inclusion of the tool on those selected pages (but notice that this will hide the tool even for users who didn't disable the tool and also for anonymous readers).

Note: If you are reading this page on www.mediawiki.org the "My preferences" link on your skin will take you to a user preferences page rooted at www.mediawiki.org which does not include the disable control mentioned above. Use the link above, or enter "My preferences" from your skin when viewing Wikipedia content.

What are the goals of this project?

The Wikimedia community cares a great deal about quality, and employs a number of mechanisms for article self-assessment (varying by project and language edition). The English Wikipedia, for example, utilizes processes for nominating and selecting the best articles, and for tagging articles associated with specific WikiProjects by quality class.

There is, to date, no standard mechanism by which large numbers of readers can easily engage in quality assessment. We believe that such a mechanism provides multiple potential advantages:

• A simple rating process could be an entry point to provide strong invitations to readers to edit, discuss, or participate in other way, as the initial evidence suggests.
• Extremely high or extremely low ratings may be useful indicators that support community cleanup or article nomination processes.
• We may be able to build upon this tool to develop a strong, standardized rating framework for content.
• Large numbers of readers who rate an article to be of high quality provide an element of external validation of our self-assessments.
• Even if the ratings are imperfect, they may reveal useful trends over time.
• An easy-to-use tool is likely to scale well to a large number of articles.

We are also experimenting with the use of an easy-to-use feedback mechanism as a way to engage readers. We are currently testing several invitations to readers after they submit a rating (e.g., edit this page, create an account) and are measuring the levels of participation resulting from these invitations. More information may be found on the project page.

How can I send comments and feedback on this project?

Feedback and suggestions to improve this tool are welcome on this talk page.

Where is it deployed?

At the moment, the Article Feedback Tool is in use on several Wikimedia wikis:

• English Wikipedia (full deployment)
• Hindi Wikipedia (full deployment)
• Serbian Wikipedia (full deployment)
• Spanish Wikipedia (full deployment)
• Portuguese Wikibooks (full deployment)
• Portuguese Wikipedia (deployed on selected articles)
• Hungarian Wikipedia (deployed on selected articles)
• Chinese Wikipedia (full deployment)

Since summer 2011, the Article Feedback Tool has been deployed on all articles on the English Wikipedia, and since november is also available on the Spanish Wikipedia. Before that, it was activated on approximately 100,000 randomly selected articles (see announcement and blog post). It was previously deployed on a smaller sample of 3,000 articles, and earlier on approximately 700 articles that are part of the Wikimedia Foundation's Public Policy Initiative.

How are pages selected for the Article Feedback Tool?

The article feedback tool can be activated on individual pages by adding the article to a hidden category:Category:Article Feedback Pilot. The activation of the Article Feedback Tool on the random sample of 100K articles was done via a "lottery" mechanism, based on the article's unique id. Every time a new page is created, MediaWiki associates a unique number to it (the page_id, whose value is preserved across edits and renames, but not deletion and recreation). This number is what the Article Feedback extension used (until the ramp-up) to determine if a page should be rateable or not. If the last three digits of the page_id of an article are in a specified range (e.g. [000,026]) the page is added to the list; If the last three digits are outside this range the page is not added. This implies that, from time to time, new pages are added to the list. E.g.: the article w:French cuisine is rateable because the suffix of its page_id is (002) is within the tool's activation range.

What are some anticipated obstacles?

We'll keep a close look at some possible issues:

• There may be attempts to game the system. We will therefore carefully analyze rating behavior in this first pilot.
• The tool may be used by people whose knowledge on the topic they are assessing is limited. We will survey the users of the system, and will also compare the rating information we will receive against other rating instruments.
• Some rating designs may reduce the incentive to edit or discuss issues. In this pilot, we will not get things exactly right, but we are well aware of this potential issue, and plan to carefully study how ratings influence user interaction as a whole.
• Low-traffic articles may receive an insufficient number of ratings, or ratings may date too quickly to be useful as articles change.
• Ratings may partially reflect the current popularity of an article's subject, rather than just the quality of the article itself.

Won't these ratings measure information a computer could easily predict?

We believe the ratings will correlate very strongly with other means of predicting the quality of an article, such as standardized readability tests. Aside from some toolserver scripts, we don't currently have any other such means, and this lack may be an area of future development and exploration.

We also think that the most interesting human-generated ratings are those that substantially deviate from what machine-derived statistics might tell us. For example, if many readers rate an article as poorly sourced, even though a computer program tells us it has many citations, that article is one worthy of further examination and possible improvement.

Do you have plans to roll it out to a larger number of articles?

As of July 2011, the tool was deployed on approximately 100,000 pages on the English Wikipedia. We continued to monitor both the performance characteristics of the feature and the value the feature provides. On July 13 we started a gradual deployment of Article Feedback Tool to the English Wikipedia. For those interested, we continue to do quantitative and qualitative research and use the results of the research to refine the tool. We also welcome members of the Wikimedia community to join our discussion to evaluate the feature and think about the future of content assessment in Wikimedia projects.

Are you suggesting that a high or low rating means that an article needs to be improved?

No. At this point, we're experimenting with this tool. It's possible, for example, that reader-driven quality assessment will only be useful on some types of articles, or will not be useful at all. We've introduced an experimental dashboard feature, which may help in assessing the usefulness of reader-submitted ratings.

How will the tool deal with multiple votes from the same account or IP (i.e., ballot stuffing)?

Only the most recent ratings from a single user are used in the calculation. If you rate an article with three stars, and then later rate it four stars, only the four star rating will be included in the calculations.

How will out-of-date ratings be handled?

Your rating will become "expired" after 30 revisions of the article. This will be denoted by a message advising you that you may wish to re-rate the article. Simply click stars again to submit your new rating.

How are the averages calculated?

Average ratings for each article are calculated based on an arithmetic average of all the ratings submitted against the last 30 revisions of the article (i.e., all "unexpired" ratings). The idea behind this calculation is to balance the need to reflect changes in the article (e.g., different revisions may have different qualities) with the fact that some articles receive very few ratings. Averaging over a longer series of revisions yields more ratings, but the ratings will correspond to more revisions, some of which may be substantially different. Averaging over fewer revisions yields fewer ratings, so the averages may be more easily skewed by ratings that reflect strong opinions (and perhaps vandalism). We will continue to experiment with different methods of showing averages and welcome discussion on the project talk page.

What's the Article Feedback dashboard all about?

We've introduced a dashboard which ranks the pages with the highest and lowest ratings over the past 24 hours. The idea behind the dashboard is to surface articles that have recently received either very high or very low average ratings, i.e., abnormal, atypical patterns in ratings that are not seen in the majority of articles being rated. We are experimenting with different ranking algorithms. It is up to the community to determine the usefulness of this information (if it is useful at all).

Currently, a dashboard of highs-and-lows is updated every hour and includes articles that have received at least 10 edits within the past 24 hours. We decided to apply the 10-edit threshold to avoid the situation where the highs (as an example) simply consist of articles that, say, have received one ratings submission of all 5's. As a result, the articles that appear on the dashboard reflect, to a certain degree, what's popular on Wikipedia within a given 24 hour period (see post) and what is receiving strange ratings (e.g., "fan votes"). We will continue to experiment with the dashboard, so please provide your feedback on the project talk page.

Is data generated by the Article Feedback Tool publicly available?

We are releasing regular weekly dumps of the data we collect via Article Feedback Tool as well as making anonymized data available on the Toolserver. Further information on the data dumps can be found on this page.

You didn't ask my permission to do this!

The Wikimedia Foundation staff has worked closely with the community, especially on the English Wikipedia, to develop this tool. This tool was originally developed in conjunction with the English Wikipedia's WikiProject United States Public Policy group (see specifically the edition of 2010-09-13 of The Signpost). A variety of discussions have taken place in multiple forums, and many people have taken the opportunity to make suggestions or share their opinions both on the English Wikipedia and off it. Responses in discussions on the English Wikipedia have generally been positive, e.g., most users here, and the community has provided some valuable suggestions and improvements in the course of these discussions. Rollouts and updates have always been announced in advance on the English Wikipedia in The Signpost (e.g., here), in the Village Pumps (e.g., here and here), and at the Administrator's Noticeboards (e.g., here) as well as in the usual channels for updates, including Wikimedia Foundation blogs and mailing lists.