Flow/New contributor survey

Liz made the excellent suggestion that we should survey new Wikimedia contributors to find out (in some detail, and with some breadth of feedback) what they need and what the use cases are.

Goal

The goal of this survey is to establish the motivations and needs of new users in engaging in discussions on Wikipedia. In particular, it focuses on:

How they understand the purpose of talkpages;
How good or bad they find the current system;
What elements of the current system work well for them, and should be preserved, and;
What elements of the current system work poorly, and should be replaced.

Target audience

The primary purpose of this survey is to gather the opinions of new contributors. Because of the deployment plan for Flow (namely, relying on small, opt-in deployments for a long period of time), there are plenty of opportunities for experienced contributors to give feedback on the designs and features, explain what they need talk pages to do, and correct our mistakes. Most of these interactions, however, will have to happen on, you guessed it, talk pages. That's fine for experienced users, but a hurdle for newcomers, many of whom haven't used talk pages before (or have had difficulties doing so).

New contributors (defined as users who have registered in the last month, made at least 5 edits during that period, and not been blocked) will be surveyed, split into two tranches - one for users who have contributed to a talk: namespace, and one for users who have not. The surveying software used will be the WMF SurveyMonkey account, with analysis performed by Oliver using R.

We have a few options to distribute the survey:

We could just drop a link on their talkpage. The problem with this is twofold; first, it increases the chance of "junk" data from people other than the intended recipient filling out the survey. Second, it undermines the neutrality of the data provided by forcing everyone to, one way or another, experience discussion pages in a highly limited way.
We can liaise with the Growth team to display a notification or message to users falling within that class for a certain time period. More of that delivery process would be automated (automated is good!) but it requires engineering time, either from Growth or Core.
We could retrieve email addresses and mass-mail people through CiviCRM. This has been done before, and does not pose legal issues.

Drafted survey

Page 1 - Fixed questions

For Tranche 1 users If you wanted to start a discussion with other users on Wikipedia, do you know where you could do that?
Yes

No
For all users Which of these types of discussions do you think are appropriate to have on a Wikipedia discussion page?
Discussing the content of articles; suggesting or requesting changes;

Discussing the contributions and behaviour of Wikipedia contributors;

Discussing the subject of articles, not just the content;

Asking for help;

Discussing things not related to Wikipedia.
For Tranche 1 users, answering "yes" to Q1 Have you tried contributing to Wikipedia's talk pages?
For Tranche 1 users, answering "yes" to Q3, or Tranche 2 users Here are a set of statements about the experience of using talk pages. Please indicate whether you agree or disagree with them. If you don't know the answer, feel free to leave the statement alone.
I find discussions easy to read.

I find discussions easy to contribute to.

I find it easy to discover when other people have replied to a conversation I am participating in

Page 2 - freeform questions

For Tranche 1 users, answering "yes", or Tranche 2 users Do you think the purpose of Wikipedia's discussion pages differs from the purpose of comment systems on other websites? If so, how?
ditto What parts of discussion pages work well
ditto What parts of discussion pages work poorly
ditto How could discussion pages be improved?

Page 3 - wrapping up

Thank you for completing this survey! If you are comfortable being contacted for follow-up interviews or questions, please provide your email address.

Resulting survey

Introduction

The Wikimedia Foundation - the organisation that operates Wikipedia - is trying to evaluate the effectiveness of Wikipedia's discussion system. As part of this, we have prepared a short survey. Please answer as few or as many questions as you are comfortable with.

By answering these questions, you permit us to record your responses for internal use so that we can make Wikipedia better. We will not share your responses publicly, except in an aggregated (non-personal) manner.

Page 1 - short-form questions

Which of these types of discussions do you think are appropriate to have on a Wikipedia discussion page?
Discussing the content of articles; suggesting or requesting changes;

Discussing the contributions and behaviour of Wikipedia contributors;

Discussing the subject of articles, not just the content;

Asking for help;

Discussing things not related to Wikipedia.
If you wanted to start a discussion with other users on Wikipedia, do you know where you could do that?
Yes

No (note: for those answering "no", survey ends here)
Have you tried contributing to Wikipedia's discussion pages?
Yes

No
Here are a set of statements about the experience of using discussion pages. Please indicate whether you agree or disagree with them. If you don't know the answer, feel free to leave the statement alone.
I find discussions easy to read.

I find discussions easy to contribute to.

I find it easy to discover when other people have replied to a conversation I am participating in.

Page 2 - long-form questions

Do you think the purpose of Wikipedia's discussion pages differs from the purpose of comment systems on other websites? If so, how?
What parts of discussion pages work well?
What parts of discussion pages work poorly?
How could discussion pages be improved?
Thanks for filling out this survey! Your responses are very helpful for us. If you would be interested in being contacted for followup interviews or questions, please leave your email address. We will not share this with third parties.

Results

Of approximately 5,000 new users contacted, 516 responded to the survey.

Short-form questions

1. Which of these types of discussions do you think are appropriate to have on a Wikipedia discussion page?

Which of these types of discussions do you think are appropriate to have on a Wikipedia discussion page?

507 users responded to this question. The vast majority of them (414) considered "Discussing the content of articles; suggesting or requesting changes" to be a valid topic for discussion pages. Other highly supported statements were "Asking for help" (306) and "Discussing the subject of articles, not just the content" (271). 193 respondents felt that discussion pages could be used for "Discussing the contributions and behaviour of Wikipedia contributors", while only 34 felt that they were for "Discussing things not related to Wikipedia."

These results are not tremendously surprising; they show that new contributors tend to have a pretty good understanding of the purpose of, for example, article talk pages, as a place to discuss the associated content, but are not as aware of meta-discussions such as user behaviour (which is unsurprising - new contributors tend not to start off in the Wikipedia or Wikipedia talk namespaces). At the same time, many assume that discussion pages can be used to discuss the subject (say, Barack Obama) rather than the content in the article itself.

If you wanted to start a discussion with other users on Wikipedia, do you know where you could do that?

516 responses were recorded for this question (it was marked as mandatory). Of those 516 respondents, only 179 were aware of where they could contact other users. This was used as a "filter" question - users who are unaware of discussion pages except as theoretical things are unlikely to provide useful qualitative information. Responses to questions after this one are therefore split into two buckets - users who answered "yes" to this one and to question 3, and "all users".

Have you tried contributing to Wikipedia's discussion pages?

Of the 179 respondents familiar with talk/discussion pages, only 69 had actively tried editing them. Again, this is a bucketing question.

4. Here are a set of statements about the experience of using discussion pages. Please indicate whether you agree or disagree with them.

Here are a set of statements about the experience of using discussion pages. Please indicate whether you agree or disagree with them. If you don't know the answer, feel free to leave the statement alone.

With the group that had contributed to or attempted to contribute to discussion pages, 73 percent found discussions easy to read (27 percent not), 71 percent found them easy to contribute to (29 percent not) and 63 found discovering replies to conversations easy (37 percent not). These are pretty promising results, although the small number of respondents makes it difficult to rely on. Data from the total group of respondents was very similar, although unsurprisingly did not contain a noticeably greater number of responses.

Long-form questions

Do you think the purpose of Wikipedia's discussion pages differs from the purpose of comment systems on other websites? If so, how?

As a long-form question, this required some hand-coding. Removing null/corrupted/non-useful responses, we had 51 responses. Of those:

10 felt that were was no difference between discussion pages and comment systems on other websites;
3 simply said there was;
18 highlighted the factual and focused nature of Wikipedia, distinguishing discussions here from discussions on other websites by a combination of (a) focus on the facts, rather than opinions (b) a desire to actively improve content rather than commenting on it and (c) a tight focus on the subject of the page rather than unrelated topics;
4 stated that users are expected to be more civil on Wikipedia than on other sites;
Other statements/concerns include describing Wikipedia as more bureaucratic than is the norm, stating that discussions are often highly disconnected and hard to navigate between, and the difficulty of confronting vast institutional knowledge.

What parts of discussion pages work well?

After removing null/corrupted/non-useful responses, we had 13 responses. Of those:

2 felt the entire system worked well;
2 highlighted the difficulty in getting responses/attention to a comment, noting that it was more of a "comment board" than a "comment and reply board";
1 felt none of the system worked;
1 stated that discussions worked so long as conversations were brief;
1 appreciated the open space;
1 liked headings;
1 liked the history page;
1 liked the behaviour of users themselves;
1 appreciated signatures - although a second user noted it would be nice if they were automatically signed;
1 appreciated the metadata through things like wikiproject boxes.

What parts of discussion pages work poorly?

After removing null/corrupted/non-useful responses, we had 21 responses. Of those:

4 found it hard to reply to comments, or to insert themselves into existing conversations;
4 found the levels of nesting difficult to handle;
3 found it difficult to find other users to contribute to the conversation or help them;
3 found it difficult to format text and write on talk pages;
2 found it difficult to find other topics, new (on other pages) or old (in archives);
2 found the tone of discussions to be unhealthy;
2 found it difficult to associate comments with the users posting them;
1 found long-winded comments made the page difficult to read.

How could discussion pages be improved?

After removing null/corrupted/non-useful responses, we had 22 responses. Of these;

5 suggested a more modern UI;
5 suggested removing or reducing the need for comment formatting;
3 suggested differentiating between different speakers more visibly;
2 suggested encouraging more replying to threads;
2 suggested actively moderating threads;
2 suggested using a thread-based, rather than page-based, system;
1 suggested encouraging civility;
1 suggested a user-based, rather than page-based, dashboard;
1 suggested notifications on replies and updates.

Conclusions

The results of the survey are ambiguous. To summarise what appears to be going on:

A majority (albeit, not an overwhelming majority) of new users find discussion pages easy to read and edit - a smaller majority find it easy to identify when conversations have been replied to;
Use of talk pages by new users is proportionately very low;
New user concerns/suggested improvements to talk pages generally align with Flow's goals and features - the elimination of wikitext as a mandatory input method, improved tracking of replies, improved distinguishing between users, and a distinction between pages and threads.

The problem with these results is, well, use of talk pages by new users is proportionately very low. The vast majority of users were unfamiliar with them, dramatically reducing the number of useful responses to the survey. A lot of respondents who were familiar with talk pages thought they were too new to contribute anything useful. Nevertheless, there were some very well thought-out and useful replies.

A suggested way forward is to take this survey into account, and to follow up with two new research projects:

Long-form interviews with particularly thoughtful respondents, exploring their concerns, needs and desires from a discussion system in more detail, and looking at Flow. It would be nice to conduct this with an editor sitting in on each one to get a look at how we do our research, ask any questions they might have, and check we're not missing anything obvious;
A second survey, this one aimed at more experienced contributors (say, registered within the last 1-3, or 3-6 months). This is being documented here.