Moderator Tools/Automoderator/Design

Automoderator design process and usability testing

Discover: Understanding the problem space

Automoderator desk research

• Some goals of the deck include:
  • Identify ethical considerations that should be taken into account at the very beginning of the project.
  • Define principles for designing a trustworthy automated system.

Design principles for Automoderator

• Design principles: 1. Transparency 2. Human-on-appeal 3. Community control 4. Ease of use 5. Timeliness 6. Bounded

Define & Develop: Designing & testing

Testing the language agnostic model

Goals: Communities tested Automoderator's language-agnostic model for accuracy. Community members could test Automoderator five 'caution' levels: very cautious, cautious, somewhat cautious, low caution, not cautious. These levels define the revert likelihood threshold above which Automoderator will revert an edit. The team compared the reverts made by the community memebrs at each level with Automoderator's reverts.

Summary of key results:

• Error rate (%) at each caution level:
  • Very cautious: 0%; cautious: 5%; somewhat cautious: 7%; low caution: 18%; not cautious: 25%.
• After the test was run the ‘Not cautious’ level was removed from Automoderator's threshold, as a 25% error rate is too low for communities.

Automoderator's pages

• 
  Automoderator's user page.
• 
  Autoderator's false positive reporting page.
• 
  Automoderator's configuration page.

User page:

The user page user page has information about Automoderator and from the page people can:

• Configure Automoderator
• Share feedback by:
  • Reporting a false positive
  • Leaving a message on its talk page

False positive page:

From the false positive page people are able to:

• Report a false positive and add it to the report log.
• Leave feedback on Automoderator's talk page.

Configuration form:

In the configuration form administrators are able to configure Automoderator for their wiki through Community Configuration.

In the form they are able to:

• Turn it on/off: Turn Automoderator on/off on their wiki.
• Configure threshold: Choose if Automoderator is most cautious, cautious, somewhat cautious or least cautious. Lower thresholds mean that more edits are reverted, but the accuracy will be lower, while higher thresholds result in fewer edits being reverted, but with better accuracy.
• Edit summary: Translate and alter Automoderator's edit summary, which is a short message that is left for others when changes are made to a page.
  • Choose if Automoderator's edits will be considered minor edits and/or bot edits.
• Talk page message: Translate and alter the message contributors receive on their talk page after Automoderator has reverted their edit.
• User groups: Ability to specify user rights whose holders Automoderator should never revert.

T358245: Tested copy, tone and language used in Automoderator's edit summary and talk page message

Contributors would be able to see and learn about Automoderator's reverts through:

• Edit summaries
• Talk page messages

Being reverted can cause frustration, so we wanted the copy in the edit summary and talk page message to be helpful and informative. We tested texts that varied in tonne, length and content to see which text was better received by people.

Goals:

• Test the UX copy for Automoderator so that we could identify any blindspots or issues with usability.
• Collect ideas and suggestions from our volunteers.

Summary of key results:

• Participants had a stronger negative reaction when the word ‘vandalism’ was used in the text.
• There was a more positive reaction to the kinder tones for the log entries but not for the talk page messages
• Participants had a preference for the talk page message as it gave them more context over the log entry.
• There was a lot of appreciation for the ability to ‘report’ and contest the revert.

T367758: Tested Automoderator's edit summaries in recent changes and page history.

Goals:

To get feedback on the new edit summary. More specifically on the copy and the false positive report link.

• Does the placement of the link interfere with existing workflows? Does this make the False positive page easily accessible? Did admins expect to find this link in the edit summary?

Summary of key results:

• People were interested in seeing a link to why something was reverted (this was also frequently requested when testing with newcomers) and an explanation of what 'unconstructive' meant.
• 50% expected the link to open the false positive page, while 50% didn't.
  • Some expected that the edit would be automatically undone when reported.

Talk page and edit summary

After testing and iterating the talk page message includes:

• Information about Automoderator reverting the edit.
• Link to the reverted edit.
• Link to article where the edit was reverted.
• Text explaining that Automoderator may make mistakes.
  • Link to its false positive page.
• Links to learn more about Automoderator and editing.

• 
  Example talk message received by contributors if Automoderator reverts their edit.

The edit summary includes:

• Link to the article, link to Automoderator's user page.
• Short explanation: Reverted a potentially unconstructive edit by [user].
  

• 
  Example of Automoderator's edit summary.