Flow/Second contributor survey

Background

This is the second survey of newcomers to Wikipedia, and their use (or lack thereof) of our discussion systems. In particular, it aims to find out:

1. How new users understand the purpose of talk pages;
2. Why they do (or do not) use them;
3. What elements of the system work well for them;
4. Where the status quo is not meeting their expectations, and;
5. How they'd improve the software to suit their needs;

Another survey ran in late 2013 with similar goals, and produced ambiguous results, largely attributable to vagueness in some of the questions and the group of potential respondents - very, very new users, a substantial chunk of whom responded with the opinion that they were too new to give useful feedback. This survey attempts to improve upon the question structure of the old one, and is aimed at a slightly more experienced group of editors.

Methodology

A list of new users will be generated. These users will be those:

1. Who have been active for at least a month;
2. Who joined less than 6 months ago, and;
3. Who have more than 10 edits.

Each user will be emailed through CiviCRM asking them to participate in the survey. (Including Flow/Survey TOS). After a week of waiting for replies, the survey will be closed and the results triaged, analysed and reported on. We may follow up with some of the more eloquent users for interviews, which I hope that existing editors might want to sit in on and participate in.

Survey draft

Page 1: basic information

1. Have you ever edited a 'talk' page? (smaller text: These are discussion pages, and feature the word 'talk' in their title - for example, "Talk:Barack Obama", or "User talk:Mr Smith", or "Wikipedia talk:Accessibility"._

   Rationale: the purpose of this is to drill down to what we actually want, which is a way to gauge how many people have successfully contributed to a talk page (or haven't!) and split both groups for successive questions. The smaller text exists to cover the possibility that people are unfamiliar with thinking of them as 'talk pages' - maybe they see them as forums, or discussion pages, or just treat their own user talk page as a 'messages' page. One risk of this approach is that people will visit the examples given, and in doing so the group who haven't contributed to talk pages will, as a result, be biased for Q2. Another thing to consider is whether to change 'edited' to 'successfully edited'; people may think of any attempt as editing. Or maybe we should just use "contributed"?

2. Whether you have contributed to them or not, how do you think people are meant to use talk pages? Tick all that apply.

   To ask for help in editing Wikipedia;

   To report problems with an article, or discuss improvements to it;

   To share your opinions about the subject of the article;

   To discuss problems with a user, or improvements they can make to their editing;

   To discuss things unrelated to Wikipedia.

   Rationale: the purpose of this question is to find out how new-ish contributors understand the purposes of talk pages. Option 1 is modified in line with Martynas's suggestions (to avoid confusion between 'I need help editing!' and 'I need help writing my homework!'), and options will be presented in a random order on a per-respondent basis. Obviously, suggestions for more options/more precise options are welcomed.

Page 2: use of discussion pages (answer yes to Q1)

1. How strongly would you agree or disagree with the following statements?

   The page's structure and user interface was easy to understand;

   Reading comments by other users was easy;

   Replying to other users was simple to do;

   Starting a new discussion was simple.

   (Options: Strongly agree, agree, don't know, disagree, strongly disagree)

   Rationale: with the previous survey we (a) offered very generalised statements (reading, editing, notifications) and (b) only offered binary options (agree/disagree). For this one we're offering more nuanced statements and a wider range of options: I've also cut out notifications, since it potentially leads us down the wrong path - if the talk page the user is familiar with is their user talk page, then they're always going to be notified. We need a better word than 'easy', though. Options for wording and more entries welcomed.

Page 2: use of discussion pages (answer no to Q1)

1. Why have you not edited talk pages?

   I tried, but it was too difficult;

   I did not know that they existed;

   I have no reason to use them.

   Rationale: with the previous survey we didn't use questions tailored to those who answered in the negative - we just threw people who had edited or tried editing into the general-purpose, long-form questions, and ended the survey for those who hadn't without inquiring as to why they hadn't. More granular exploration of their experiences will allow for page logic that lets us find out more information and tailor question structure in a precise manner. As with the other questions, I'm open to more possible answers here.

Page 3: improvements to discussion pages (Yes to Q1)

• Based on your experience editing talk pages:

1. What bits of the process or page were easy to understand and use?
2. What bits of the process or page were difficult to understand and use?
3. What could we do to improve how talk pages work?

   Rationale: with the previous survey we kept things very general - too general. It was potentially difficult for respondents to understand if we were asking about the users, or the interface, or the process, or all of the above. This seeks to break that down a bit more, although I'm worried that it may still be too vague - comments are appreciated.

Page 3: improvements to discussion pages (No to Q1, "too hard" to Q3)

1. What part of your experience trying to edit talk pages did you find difficult?
2. How could we improve these?
3. What parts worked well?

   Rationale: very rough, but similar to the other Page 3; the differences are a change in focus (respondents are biased towards those who found things difficult, and so prioritising an explanation of what they found difficult is worthwhile) and changes in wording along with that. Again, rough, ready, needs improvement.