Wikimedia Apps/Team/Android/Anti Vandalism

Background edit

The Android team has received requests from various language communities to improve the quality of edits made through the app. Community members have asked for moderation tools that are fair and unbiased across language wikis. The team aims to enhance existing workflows while ensuring equity in moderation, an example can be found in the 2022 wishlist.

With recent updates to the app, such as the native watchlist, contribution history, and edit history, as well as the addition of the undo and rollback button on the diff screen, there is an opportunity to create a moderating solution within the app. The solution will build on existing mobile micro contribution workflows.

The team plans to develop an anti-vandalism tool (V1) that caters to experienced moderators across languages. Additionally, they will introduce a training task to help experienced editors learn patrolling if they haven't done it before. This will be done in close collaboration with the community. The team will also work with the research team to improve tools like ORES and collaborate with the Moderator tools team, who are addressing this issue on Mobile Web, to ensure a seamless experience for cross-platform users.

Product Requirements: edit

  • Editors can review diffs for a variety of edits
  • Editors have the option to 'revert' or 'skip' suggested changes
  • Recent edits made in various languages can be displayed
  • Editors can create and save warning messages to use when reverting changes
  • Experienced editors will be introduced to patrolling on the app in a later version (V2)

Technical Limitation edit

Currently, it's not possible to have multiple languages in a single feed, but users will have the ability to switch between languages.

Hypothesis & Validation edit

Hypothesis & Assumptions edit

We believe that simplifying the patrolling tasks and structuring them as advanced-level tasks would be beneficial. Experienced editors without sysops or rollback rights could be assigned easier patrolling tasks or a training task, and once they reach a threshold of initial patrolling activities, they can be promoted to higher levels. To achieve this, we plan to start with a proof of concept for general patrolling of edits based on users with rollback rights.

We understand that patrolling recent edits is a priority and more suitable for mobile than tasks like NPP (New Page Patrol), AFD (Articles for Deletion), and AFC (Articles for Creation). Additionally, we expect that there will be community consensus on implementing a training task across different languages.

Validation edit

  • Key Indicator 1: 65% of target mature audiences using the tool find it helpful for maintaining the quality of wikis and would recommend it to other patrollers
  • Key Indicator 2: Increase in edits made by mature audiences by 5%
  • Key Indicator 3: 10% of target mature audiences engage with filter preferences
  • Key Indicator 4: 65% of target mature audiences engage with the tool at least three times in a thirty-day window
  • Guardrail: Experienced users without rollback rights, both those who have and have not used alternative patrolling tools, should equally understand the workflow
  • Guardrail: There should be no reports of the tool being used to negatively target underrepresented content or contributors based on in-app reporting mechanisms

Things that would be helpful to know

  •   How does use of our tool compare to other patrolling tools when looking at MediaWiki Tags (SWViewer, Huggle, and Twinkle)
  •   Do we see an increase in Undo/Rollback/Thank events
  •   How popular is this task with our target audience relative to other Suggested Edits task?
  •   What actions are most popular in the feature?

Risk management edit

Risk Cause Level Response Response Action Trigger Contingency Plan
Tool causes a decrease in new editor retention on wikis where it's available Editors use the tool to increase the volume of edits they're able to revert, but do not leave talk page messages to explain the revert, leaving new editors confused and unmotivated. Medium Monitor Measure the retention rate of editors who had a revert from the patroller task. If they got a talk message, did that increase their retention? Qualitative complaints If a user has not left a talk page message after using undo or rollback and attempts to move to next edit, add a prompt asking/requiring them to leave a Talk Page message
Policy disagreements between existing patrollers and newer patrollers using Edit Patrol / New patrollers use the tool a way that confuses other patrollers. Behavior and/or communication from App patrollers is seen as insufficient or lacking by other patrollers. A new group of editors are able to start patrolling at scale using Edit Patrol, but they are not aware of existing policies or norms. Medium Mitigate Post updates, and invite to office hours patrolling groups (CVU, recent changes patrol, small wiki monitoring team, stewards, global sysops, global rollbackers) before full release. Share Example Messages with communities. Qualitative complaints from patrollers Add more policy and template onboarding to tool, and revisit Example Messages after Community Configuration 2.0 is released. Continue work on building training task.
An increase in edit wars Patrollers reviewing the same feed of edits go back and forth reverting each other's changes because revisions or rollbacks made in the tool appear in the feed for the next person (if no filters are applied). Low Accept - Qualitative complaints Add more onboarding about policies surrounding Edit Wars. Add in-app warning for edit-war behavior.
The tool is being used to negatively target underrepresented content or contributors. Patrollers are systematically reverting good faith edits for particular users or languages, harassing, or mistreating contributors through the tool. Low Mitigate Unless otherwise requested per wiki, the tool available to those with rollback rights. Qualitative complaints, in-app reporting mechanisms, guardrail analysis by Product Analyst Add universal code of conduct onboarding to tool.
The tool causes an increase in vandalism, and increase in amount of work for existing patrollers Vandals use the streamlined interface to quickly revert or rollback large quantities of good faith edits. Low (Dormant) Mitigate Unless otherwise requested per wiki, the tool available to those with rollback rights. Qualitative complaints and anomalies surfaced in data reporting Implement a maximum number of undos/rollbacks that can be done in a range of time.
Someone is exposed to vandalism or there is a copyright violation in Example Messages provided in-app A vandal has edited the Example Messages or pasted copyright material in using Community Configuration 2.0 or Translate.wiki, and they automatically appear in everyone's app Low (Dormant) Mitigate Once community configurable, ensure permissions are set for editing with Growth team. Qualitative complaints Inrease necessary permissions to edit Example Messages.

How to Follow Along edit

We have created T322083 as our Phabricator Epic to track this work. We invite you to collaborate with us there or on our Talk Page.

We will provide periodic updates on this page as we make progress. If you are interested in participating in our moderated design feedback sessions, please let us know on our talk page, and we will follow up with you.

We have the following roll out strategy:

2023–2024 time period Consultation Released to rollbackers
October 2023 Indonesian Wikipedia
November 2023 Indonesian Wikipedia Indonesian Wikipedia
January 2024 Spanish Wikipedia, French Wikipedia
February 2024 Spanish Wikipedia, French Wikipedia
March 2024 Chinese Wikipedia, English Wikipedia, Igbo Wikipedia Spanish Wikipedia, French Wikipedia
April 2024 English Wikipedia, Igbo Wikipedia, Global Sysops and Rollbackers English Wikipedia, Igbo Wikipedia
May 2024 TBD Available for all wikis

Technical Documentation edit

If you are interested in seeing how the feature is implemented, please visit the Patroller Tasks Dev Note for more details.

Updates edit

February 2024 edit

  • We've started development on Saved Messages T356774 (example messages available in the App) and access to templates T355141, which was well received by French Wikipedia. Draft saved messages are posted here, and we welcome feedback on the discussion page.
  • Another community member raised interest in the example warning messages being configurable via the community config.(T358265#9573393), and we are coordinating with the Growth team on this.
  • French Wikipedians reaffirmed the choice of Edit Patrol only being available to rollbackers.

January 2024 edit

  • Indonesian Wikipedia requested consideration for using Edit Patrol in the upcoming election to fight disinformation and vandalism. Our team had the feature reviewed by our legal team, and volunteers were cleared to use the tool for patrolling during their upcoming election.
  • This month we learned that some communities appreciate being able to have speedy deletion as an option in patrolling tools, which we will consider for future iterations of this feature.
  • Additionally, our team met with the Growth, Moderator Tools and Machine Learning teams to discuss adding the revert risk model to recent changes filtering. We may be able to switch over from using Good Faith and Damaging ORES scores to the Revert Risk model as we scale to Spanish, French and Chinese Wikipedias in the coming months. This may open the door for an experimental training task for experienced editors without rollback rights.
  • We have established demo sessions for our next set of pilot wikis:
    • French Moderators - scheduled for 21 February 2024 from 16-17 UTC
    • Spanish Moderators - scheduled for 28 February 2024 from 16-17 UTC
    • Chinese Moderators - recorded presentation to be shared asynchronous
  • Our designer worked on the flow and UI for the saved warning messages and templates, which will be released in the coming months.
  • Template Flow: T355141: The team is working to add support for templates, so patrollers can utilize existing templates in their messages

December 2023 edit

Patrolling Pain Points on French Wikipedia

We met with a few language communities to get feedback about Edit patrol. The Indonesian Wikipedia community approved the release of the feature for their community and shared which templates are used for warnings, which we will incorporate in a future release. For the month of December we are enabling users to save warning messages on their device (T343841). In January we will begin outreach to Spanish and Chinese Wikipedia to scale to more communities. We’ve already begun outreach to French Wikipedia via the French Wikimedia ED, he shared some pain points identified by French patrollers as it relates to RC Live and SWViewer and other tools as it relates specifically to the French Wikipedia community that we are taking into consideration for Edit Patrol. We are hoping to establish office hours sometime in February with French Wikipedia to discuss Edit patrol.

November 2023 edit

Release to Indonesian Wikipedia

Edit Patrol was released to Indonesian and Test Wikipedia. In order to see the feature in Suggested Edits you must have rollback or sysops rights.

Through this first release, rollbackers are able to:

  • Onboarding to the task with an explanation of when to use Thank, Watch, Warn, Undo or Rollback, as well as an explanation of ORES scores. (T343242)
  • Compose, save, and publish first warning messages. (T344842)
  • Select, edit, save and publish existing warning message (T344842)
  • View information about other editors including their tenure as an editor and their edit count (T343835)
  • Filters for the recent changes feed include language wiki, registration status, experience level, quality score prediction, user intent prediction, automated contributions and significance of edit. The filter page also allows sorting by latest revision. It is also possible to search through edits.
 
Filter with no results
  • An ability to report issues with the filter (T343834)
 
App bar menu-problem with the feature dialog

We welcome feedback about the feature as we prepare to reach out to additional wikis to scale the feature.

September 2023: Advancing V1 of Patroller Tasks: Unveiling New Features to the Indonesian Wikipedia Community edit

We continued development on V1 of Patroller tasks and announced the feature to Indonesian Wikipedia. Our engineers demoed the filters for Patroller task V1:

August 2023: Project Updates and Technical Considerations edit

  • We shared updates about this project at Wikimania. The recording can be found on Youtube starting at 1:24:30. We received a question about having a central place for Warning messages that can be used across apps. We are planning to work with the Growth team to explore the possibility of centralized warning messages.
  • We became aware of the deprecation of ORES and migration to Liftwing which will host the revertrisk model. Due to our team using the MediaWiki API to show the recent edits feed, we will continue to show the Goodfaith and Damaging Models at least in V1. Using the Revert Risk model would require two API calls, which could slow down the feature, this also doesn’t address filtering. If you have feedback about this technical decision, feel free to comment on our discussion page.
  • We created designs for Onboarding to the feature T343242.

July 2023: Design Research and Interview Results edit

Usability Testing edit

A moderated remote usability test was conducted involving six experienced contributors who use Android devices and volunteer their time for monitoring activities on Spanish, French, English, and Chinese language wikis. We selected participants from these wikis due to their high number of editors with sysop/rollback rights, which allowed us to gather usability feedback effectively.

The patrollers provided valuable insights on early designs and the usability of the 'Edit patrol' tool.

During the test, the editors were tasked with completing short assignments using a limited prototype of the tool. They were also asked to provide real-time narration of their experience as they worked through these tasks.

For additional details about the usability test, please refer to the following link.

Results edit

The feedback and valuable suggestions from participants have contributed to enhancing the initial designs. The following improvements were implemented:

  • Participants expressed the desire to have tags displayed for each edit on the recent edits page, and the ability to discern if an editor made multiple edits to a single page.
  • Our goal was to incorporate filters on the page where recent edits are showcased. To determine the filters for the initial version of the 'Edit patrol' tool, we consulted with the patrollers. They suggested including the following filters: languages, user registration and experience, automated contributions, contribution quality predictions, user intent predictions, latest revision, and significance. Subsequently, we designed a page that features these filters, which is presented below.
  • The diff view page was simplified
  • Minor design enhancements have been applied to the process of creating, saving, publishing, and editing warning messages for patrollers. The revised designs are displayed below for your reference.
  • Participants provided further feedback regarding patrolling, which we aim to convey within the tool via onboarding, tooltips, or reminders. The points include:The significance of communicating with empathy to fellow contributors, and the emphasis on prioritising the quality of patrolled edits over their quantity.

The 'Edit patrol' tool is currently in the development phase, and you can track its progress through Phabricator T343224.

February 2023: Initial Design Concepts edit

The team developed initial design concepts by incorporating insights from existing tools and feedback we gathered on their limitations. We shared these concepts with the Moderator Tools Team and Research Team for their input and feedback.

We will use the feedback received to improve the design concepts and establish a protocol for both moderated and asynchronous feedback from app users. Currently, we are initiating direct outreach to schedule moderated feedback sessions and ensure translation mechanisms are in place for those sessions. Once the protocol is available on MediaWiki, we will send out a broader communication to start gathering asynchronous feedback on our talk page.

The design concepts provided are meant to offer examples of how we aim to address the issues faced by other tools. However, please note that we are in the early stages of this process, and the designs will evolve and expand based on the feedback we receive in the coming weeks.

January 2023: Comparative Review and Existing User Research edit

The team conducted a comparative review to analyze the existing patrolling tools. During the review, we assessed the similarities and differences between the tools and identified which ones were effective on mobile devices.

The complete list of tools we reviewed include:

  • Huggle
  • Twinkle
  • TwinkleMobile
  • STiki
  • SWViewer
  • WikiLoop Double Check
  • CheckWiki
  • ReWarn
  • CheckWiki

Through our evaluation, we found that the majority of tools were optimised for Desktop use, with only three exceptions. Additionally, only two of the above solutions performed well across various language wikis.

Common elements found in these tools were:

  • Icons to represent the main action (e.g. revert, speedy deletion, welcome user, etc.)
  • Access to templates
  • Filters
  • List of contributions
  • Diffs
  • Information about the editor

In addition to the comparative review of existing tools, we also collected recurring requests from patrollers regarding those tools. The common requests include:

  • Filter contributions by topics of interest and specific time period
  • An easy way to exclude contributions that have already been patrolled
  • No requirement for special privileges like rollback rights to access the tools
  • An option between "skip" and "revert" for more nuanced actions
  • Tools that promote collaboration among patrollers

By taking these requests into account, we aim to develop a comprehensive patrolling solution that addresses the needs and preferences of patrollers across various languages and user experience levels.

December 2022: Proof of Concept Prototype edit

The team developed a proof of concept prototype to explore the capabilities and possibilities using existing APIs. The prototype allowed us to test and experiment with different functionalities and assess the feasibility of our ideas. This early exploration helps us understand the potential of the tools and refine our approach before moving forward with further development.

During our development process, we successfully exposed the ORES score, which provides users with an indication of the likelihood that something may be vandalism. This feature allows users to get insights into the quality of edits and helps them make informed decisions during the patrolling process. We would ideally send feedback to the research team's API for improvements, which will be valuable in improving the accuracy and effectiveness of the ORES score.

November 2022: Defining Initial Audiences edit

The team conducted research (T322065) to assess the number of Wikipedia app users with sysops and rollback rights. The findings revealed that 372 Android app users and 176 iOS users had these rights. Based on this data, we decided to prioritise the rollout of the feature in the Android app, allowing the iOS development team to focus on building the Watchlist feature.

Additionally, we sought to understand the distribution of users with sysops and rollback rights across different languages. This information played a crucial role in determining the initial target language communities for our direct outreach during the initial research phase. By focusing on these specific language communities, we aimed to gain valuable insights and feedback from users who have experience in moderation and patrolling, which will be instrumental in shaping the development of the feature for wider use:

  • English Wikipedia
  • French Wikipedia
  • Chinese Wikipedia
  • Spanish Wikipedia

While we are focusing on direct outreach to specific language wiki communities and OS users, we want to emphasise that we welcome and highly encourage feedback and ideas from any app user with patrolling experience. We value the input of Web users as well, but our priority will be to gather feedback from app users.

It is essential to create a unified experience across platforms, and to achieve this, we are in close communication with the Moderator Tools team, who will be building experiences for Web users. By learning from their findings and collaborating closely, we aim to ensure continuity and consistency across both the app and Web platforms. Your feedback and ideas will play a crucial role in shaping the development and improvement of the patrolling feature for all users.