Flow/Research

See more recent research at Flow/Moderated Testing, November, 2014: talk pages and Flow.

Research on talk discussions has been a long-running process.

New Users

Flow prototype testing

Flow UX workshop, October 15, 2013
Usability testing, November 23, 2013

Test Videos

The Foundation conducted several user tests to get a glimpse into the problems facing new users when they are exposed to what are relatively simple (and common) scenarios with user talk pages.

Tests have been annotated, calling out specific issues that users face and behaviors that they expect.

Raw data and links to view individual tests may be found at User Test Data.

Takeaways

None of the tested users were able to intuitively grasp anything about the use of User_talk pages.
On average, it takes new users around 15 minutes to grasp the basics of user talk messaging.
Users generally have negative impressions of using wikitext for communication ("it's like math"). It is extensively confusing and somewhat demoralizing.
Users have expectations that the discussion system will be significantly more modern and be much more user friendly. To wit:
- Users have difficulty locating their new messages and talk pages.
- Signatures and colon-indentation are especially confusing.
- Users expect an obvious mechanism for replying to comments. They do not understand section edit links.
- Users do not generally understand how to watch for replies (or if this will even happen).
- Users have an expectation that those they engage with will be notified in a sane manner.
- Users expect a way to leave messages for other users from User pages (not just User_talk).
Talkback templates are completely confusing and non-intuitive.
Users are surprised that they can edit other people's messages.
Users have difficulty understanding that conversations can exist in multiple locations.

Note that the scenarios that these users are sent through - and the talk pages they end up on - are typical experiences for new users.

Experienced Users

Feedback during first release

Below are charts showing the sentiment of feedback in various feature areas that users gave during the early "sandbox" release period. This feedback was gathered from Flow-enabled pages on the ee-flow test wiki and mediawiki.org during November-December 2013; it was qualitatively hand-coded for feature area and sentiment.

Note: this is a very small sample (only 28 unique users total gave feedback during this period), so the results should not be considered representative of all experienced users.

Quora Question

User:Jorm (WMF) asked on Quora: What do you find confusing about Wikipedia's discussion systems?. The full text of the answers can be found at Experienced user responses.

What do you find confusing about Wikipedia's discussion systems?

Not everyone is aware that there actually *is* a discussion system on Wikipedia. There are two basic types: discussions about articles and then per-user discussion spaces (these are called "Talk pages").

I am currently working on a design proposal to build a better discussion system. I know what I think is confusing about Talk pages (there's a big list) but I want to hear what other people think, too, so that I can fold issues I may have missed into the design.

Experienced User Suggestions

These are suggestions from experienced users for improving the user-to-user discussion system. This list is by no means exhaustive, and is the result of a limited set of answers.

Bugs/Issues

Too much wikitext:
- Signatures should be automatic
- Indenting is tedious
- Managing these errors by inexperienced users adds workload to experienced ones
- Wikitext is "too free"
The label "talk" is counterintuitive.
- It doesn't stand out enough that there is a conversation space.
No way to watch specific threads; must watch entire page.
Archiving is completely broken in many ways.
- Lack of permalinks is especially bad
- Context and history in conversations is typically lost
Where do replies go? My talk page or someone elses?
Lack of notification when someone mentions you in a discussion or replies to a discussion you are involved in
Timestamps can be difficult to follow or clutterish
Thread/topic management is an issue.
- To easy to distort conversations through refactoring
Talkback templates/notices add clutter
On-wiki processes (such as suggesting merges or deletions) are complex and difficult to follow

Feature Requests

A desire for context/annotations
Namechecking/mentions (@Username)
Better way of linking to diffs
Native support for quoting
Profile photos or avatars. Discussions can be hard to follow and lack empathy
A way to "upvote" comments or indicate simple gratitude or support

Past research on user talk

The shift in the positive effect of receiving a talk page message from 2003 to a negative effect from 2006 on

First messages to new users by type, 2004-2011

Sample of new user help requests, 2011: the majority happen on user talk pages

"Ignored" new users

Since 2006, new users who are "ignored" (e.g., do not receive any talk page messages) are more likely to keep editing than users who do receive a talk page message. This suggests that the current way we communicate to new users is doing more harm than good.

However, for new users whose edits are reverted, not sending a warning is much more likely to make the user go away forever than sending a template warning. 41% of new users will continue editing after being sent a template warning for a test edit, versus 9% who will continue editing if reverted and sent no warning.

Documentation (1, 2)

First messages to new users

Since 2006, English Wikipedia has seen an explosion of semi-automated tools and bots that send welcome and warning messages to new users. By 2011, 80% of first messages to new users were templates sent by a tool or a bot, and 65% were warning templates.

Documentation

The impact of user talk templates on new users

About 50% of registered and unregistered users check their talk pages within 24 hours of receiving a message. The majority of users whose edits are reverted and who receive a warning never return to edit again. Changing the tone of template warnings has a small but significant effect on short-term retention, but not long-term retention.

Documentation

New user help requests on user talk

Based on a sample of new user edits from 2011, the majority of new editors ask for help on user talk pages – their own or another editor's.

Documentation