Flow/Community process workflow interviews (June 2015)

Group:	Design Research
Start:	2015-06-15
End:	2015-06
Team members:	Danny Horn, Daisy Chen,
Backlog:	#WMF-Design-Research
Lead:	Jonathan Morgan

Flow is an alternative to the current talk page format under development by the Wikimedia Foundation, which aims to provide a cleaner talk page experience/interface for readers and editors on Wikipedia and Wikimedia sites. The Collaboration team is beginning to design a Workflows feature for Flow -- a key feature for the more experienced Wikipedia contributors. We need to check in with a group of experienced editors early in the design process, to make sure the team is going in the right direction.

Summary of findings

Preliminary findings from a study of community processes on Wikimedia wikis performed to inform the design of Flow.

while workflows are idiosyncratic, many follow a general Proposal -> Discussion -> Resolution pattern
specialized tools (bots, gadgets, templates, userscripts) have been developed to support the needs of individual processes
key challenges for participants include starting a discussion, notifying relevant people and projects, and tracking discussion progress
key challenges for process leaders (closers, moderators, admins) include decision-making and backlog triage

Summary of recommendations

Provide configurable notifications for each conversation pattern to help users track discussion progress.
Provide fine-grained configuration options for the structure of each conversation pattern, including standardized sub-headings, !vote tags, character limits, announcement templates, and closing summaries.
Provide features that allow users to monitor multiple discussions concurrently.
Provide features that help closers make decisions more efficiently.
Provide features that help closers triage open discussions to reduce backlogs.
Pilot test initial Workflow products with less complex community processes that involve lots of discussion and have fewer specialized rules and roles.

Avoid designing features that encourage people to !vote rather than have a conversation.
Avoid artificially limiting who can perform certain actions within a discussion--keep everything open by default.
Avoid breaking existing gadget- and bot- mediated discussion archiving mechanisms.
Don't assume that processes with similar names/functions on different Wikimedia projects work the same way.

Goals of this research

The goal of this round of research, to be completed in June 2015, is to hold interviews with prospective users of Flow’s proposed Workflows feature.

The purpose of this research is twofold: (a) to learn how Wikipedia editors participate in community processes, what problems they experience, and what would help them in their work; (b) to find out whether our current ideas for a Workflows feature would help them reach their goals.

Specific goals

Understand the mental model of Wikipedia-related workflows (how do they normally participate).
Road-test the underlying logic with a knowledgeable user:

Is this definition of workflows easy to understand?
Is it able to capture the kind of workflows you know or can imagine on Wikipedia?
Is the way to execute workflows easy to understand, and would this be a more efficient process than the current system?

Identify opportunities to improve our concepts and thinking.
Get new perspectives and suggestions.

Key terms

Community process: Any discussion that starts with a proposal, has some specific rules/structure, and ends with a decision.
Workflow: The steps involved in any formal, multi-step community process.
Conversation pattern: A Flow board that is configured to support a particular kind of workflow.

Timeline

Dates	Activities	Status
Phase I: June - July	Create initial mock-ups, conduct first round of interviews	Complete
Phase II: July - August	Create additional mock-ups, conduct second round of interviews (focused on notifications)	not staged
Phase III: August - ?	Test workflows with on test wikis and on live wikis with groups that have volunteered	not staged

Methods

We conducted interviews with Wiki[m|p]edians who participate regularly in one or more community process (such as Articles for Deletion, Administrator’s noticeboard, Good Article review, Dispute Resolution). The goal is to get as much "coverage" as possible with a relatively small set of participants (5-10), so we sampled for diversity.

In the first round, we interviewed seven participants about 14 processes across English Wikipedia, Meta, Italian Wikisource, and Wikimedia Commons. The interviewees are very active contributors on Wikimedia projects (more than 1,000 edits).

We used a semi-structured interview protocol to learn about the basic mechanics of the process, its purpose and relative importance, the context in which in occurs, who participates, etc. After the participant has described the process thoroughly, we introduced them to some early-stage prototypes and wireframes for the proposed Flow workflows interface that showed how Flow might be configured by editors to support a specific process, and how the subsequent discussion could work.

Interview protocol

Click to view protocol

This is a semi-structured interview protocol. Not all questions will be asked in every interview, and the investigator may ask additional probe questions if interesting things come up.

When possible, the investigator will ask the participant to share links to pages that they talk about or diffs that point to particular actions or events which are referred to over the course of the interview. The investigator will also encourage the participant to share their screen as they describe their workflows.

Part 1
First, I'd like to get a little information about who you are and what you do on Wikipedia

What kind of work do you do on Wikipedia?

Now I'd like to learn about the community processes you participate in most frequently

which community processes have you participated in most during the past year?
how do these processes relate to other work you do?

Now, let's talk about a process that you've been heavily involved with recently.

tell me a little bit about the process and its goals
how many people participate?
In a normal week, how many of these discussions are in progress?
where does discussion take place?
how would someone who had never participated in this process learn that it exists? How would they learn how to participate?

Tell me a little bit about how it is organized

under what circumstances is this process initiated? who initiates it?
how long does it take before resolution?
do people have specific roles in this process? tell me a little bit about how people in different roles participate in the process.

Now I'd like to learn a little more about how you participate in this process. Thinking about the 'work' you do...

what role do you generally play in the process?
How do you decide whether to participate in a new discussion?
How do you keep up to date with what's going on?

Now I'd like to hear about what you think about the process in general.

Tell me about a recent fun or satisfying experience you had while participating in this process
Tell me about a situation where you were frustrated while working participating in this process
How successful is this process? How do you measure the success of a particular discussion? Of the process as a whole?
How important is this process to Wikipedia? Why?
Are there specialized tools, like userscripts, gadgets, or templates, that are used in this process?
Is there any tool that you use that you wish worked differently or better? How would it help you?
Is there any tool that your project doesn’t have that you wish it did? How would it help the project?
Are there improvements that would be especially useful for newcomers? For experienced editors?

Part 2

Walk through the Conversation Pattern prototype and ask participants how it might work for the processes that they participate in.

Basically, from here, we want to have a conversation with the user. We’ll have several concepts and mockups, and possibly a clickable prototype.

We want to show each artifact that we have, and at different points, ask these questions:

Does this piece make sense? Can you imagine how you might use it?
Is there anything that jumps out at you that looks exciting?
Is there anything that just feels wrong, and you don’t think it’ll work?
Can you think of a workflow (or a type of workflow) that wouldn’t work with this model?
Do you have suggestions?

How to explain the prototypes

The Flow team could make (for example) the Articles for deletion workflow in software. There’s a place to nominate the article, and it automatically fills out the user and the article name, prompts for a reason, puts a template on the article and then lists it on the AfD list. That would be relatively easy to build. But -- there are a thousand different workflows, some of them change over time, and there are different versions on different languages. So what we have to build is the set of tools that helps experienced editors to create their own workflows, for other people to use.

Prototypes

Click to view list of prototypes we used

Conceptual prototype (AfD)

(developed before research began)

Workflow configuration mock-ups

(developed during phase 1)

Workflow discussion mock-ups

(developed during phase 1)

Findings

Community processes map

Process overview

Process	Project	Discussion location	Discussion scale	Who gets notified	Supporting tools used	Participant
Articles for creation (AFC)	EnWiki	content page in Draft or User namespace	small	author, article	AFC helper gadget	p5
Articles for deletion (AFD)	EnWiki	Portal subpage	medium	WikiProjects, primary contributors	Twinkle, various Deletion sorting userscripts	p5
Deletion requests (DEL)	Commons	Portal subpage	small to medium	uploader	AjaxQuickDelete, DelReqHandler (gadgets)	p2
Featured Article Candidate (FAC)	EnWiki	Portal subpage	large	article talkpage	archiving bot	p3
Featured Article Review (FAR)	EnWiki	Portal subpage	large	primary contributors, FAC nominator, WikiProjects	archiving bot	p3
Peer Review (PR)	EnWiki	Portal subpage	medium	WikiProjects, article talkpage, related article talkpages, PR volunteers		p3
Request for Comment (RFC)	EnWiki	Main, Wikipedia, Template, etc. namespace talkpage	large	feedback request service subscribers, article watchers, relevant noticeboards, relevant WikiProjects, talkpages of related articles	archiving bot	p1
Request for Closure (ANRFC)	EnWiki	RfC location	n/a	n/a	none	p1
Administrator Intervention Against Vandalism (AIV)	EnWiki	Portal page	small	reported user	Twinkle, userscripts	p4
Administrator’s Noticeboard - Edit Warring (ANEW)	EnWiki	Portal page	small	reported user(s)	Twinkle, userscripts	p4
Dispute Resolution Noticeboard (DRN)	EnWiki	Portal page	large	all 'interested' editors	DR wizard (gadget), archiving bot	p6
Steward Requests for Permission (SRP)	MetaWiki	Portal page	small to medium	none	archiving bot	p2

Process description

Process	Workflow
Articles for creation (AFC)	An editor, usually a newbie, creates an article. Could be in the main, Draft, or User namespace. Welcome templates often contain links to the Article Wizard, which helps the user create the article page. Any editor with sufficient edits can 'claim' a review by putting a template on the draft page. The reviewer provides feedback and a decision in the template, and may add additional comments, then informs the author of the decision on the author's talkpage. An article can be re-submitted for review at any time, if in Draft or user space. The AFC helper script gadget is extensively used by reviewers to manage the process.
Articles for deletion (AFD)	Nominator creates a page under AFD for the discussion, adds a template to the top of the article, and notifies related WikiProjects and substantial contributors to the article. Deletion discussions run at least seven days, during which editors vote to keep, delete, etc. Any editor can close the discussion after consensus has been determined, but certain closing actions (such as deletion) require administrators. Deletion discussions are archived by topic for future reference by WikiProject Deletion Sorting.
Deletion requests (DEL)	Nominator places a notification template on the File page and on the uploader's talkpage, and creates a subpage on the deletion request noticeboard where the discussion will take place (this is usually aided by a gadget). Thread is left open for at least 7 days, after which it is closed by an admin with a decision like keep, delete, no consensus, etc.
Featured Article Candidate (FAC)	Nominator places a template at the top of the article talkpage, and creates a FAC portal subpage to host the discussion. Other editors can vote to support or oppose the promotion, and may offer more elaborate reviews of the article with suggested changes by creating subheadings like "Review by User:Foo". The nominator attempts to address the reviewer's changes and coordinates with them within that sub-thread. When the nominator is ready, they notify one of the FAC coordinators, who makes a determination to promote the article, or not. The article talkpage template is updated by bot with the outcome.
Featured Article Review (FAR)	Nominators are expected to raise any issues on the article talkpage before submitting a Featured Article for review. Subsequently, they add an 'under review' template to the article talkpage, create a FAR portal subpage for the review discussion, and notify interested parties of the discussion. The first step of the discussion involves proposing improvements to the article (similar to the FAC process). Any editor can act on these suggestions; the nominator is encouraged to do so. If no consensus is reached about the listing status of the article, the discussion moves to a voting stage. The FAR Coordinators make a final determination, based on the arguments presented. Each stage typically lasts several weeks. The article talkpage template is updated by bot with the resulting decision.
Peer Review (PR)	Requester places a template on the article talkpage and creates a PR portal subpage where the discussion will take place. The requester can ask any editor to review, but the instructions suggest notifying WikiProjects and/or editors who have listed themselves as volunteer reviewers for articles within a certain topic. Reviewers typically create discussion sub-sections for their reviews. Requesters can coordinate with them directly: asking them to re-evaluate the article after their feedback has been incorporated, for example. The requester can close the discussion at any time. Any editor can close a peer review that has been inactive for a long period of time. There is no decision component. Reviews outcomes are linked from the template on the article talkpage.
Request for Comment (RFC)	Requester places a template on the article talkpage, specifying the subject area of the RfC. The requester adds a neutral statement of the issue at hand directly below the template. A bot posts notifications/transclusions in the proper places. Requester is responsible for posting notifications on relevant boards and project portals. Any editor can comment and indicate their support or opposition; the result is determined based on the quality of the arguments and/or degree of consensus, not the number of !votes. The default duration is 30 days, but any uninvolved editor can close the discussion if consensus has been determined, or formal closure can be requested at ANRFC. A summary of the decision and rationale is posted at the top of the RfC.
Request for Closure (ANRFC)	If there is no clear consensus on an RfC after 30 days, or if it has wiki-wide implications, editors can request closure at the Administrator's noticeboard. The requester should add a neutral description of why they are asking for special closure. Any editor can still close the discussion (under most circumstances), but mostly admins do the work. If an editor thinks that an RfC was closed incorrectly, they can ask for admins to review and potentially overturn that closure via this process, as well. Formally closed RfCs are archived in a separate archive.
Administrator Intervention Against Vandalism (AIV)	Any editor can report vandalism via AIV by adding a new templated request form to the top of the AIV board. A sample template is available in the edit window, for editors to copy and customize to their request. The template includes the alleged vandals username, and a brief justification for the request. Any administrator can make a determination of how to close the request, and there are many options--from "page deleted" to "insufficient activity to warrant a block", to "user warned". The administrator then takes the action appropriate to their determination: blocking a vandal, or warning a user on their talkpage, for example. In most cases, these requests involve little discussion, and few participants. Administrators who frequently patrol AIV use Twinkle and other userscripts to make the process more efficient.
Administrator’s Noticeboard - Edit Warring (ANEW)	Similar to AIV, except that reporters must notify the user they are reporting on the user's talkpage, and there are generally more people involved in the discussion (including uninvolved editors besides the closing admin). Reporters are required to provide diffs as evidence of 3RR or other edit war violation, as well as previous attempts to resolve the dispute. Twinkle provides workflow support for closers. Decisions could include blocking/warning users, or protecting pages where the edit war is occurring.
Dispute Resolution Noticeboard (DRN)	A user creates a new thread using the Dispute Resolution Wizard gadget, which elicits information such as the location of the dispute, the users involved, and a summary of the nature of the dispute and of previous attempts to resolve it. This information is added at the head of the new thread which is created. A DR volunteer then 'claims' the case by opening the discussion with a statement and/or questions for the parties involved. Those parties then make 'formal statements', followed by the volunteer moderator's response. This can go on for several rounds. The moderator suggests resolutions and makes a non-binding determination w/ a closing statement to end the discussion. A bot updates a dashboard of open moderation discussions throughout the process.
Steward Requests for Permission (SRP)	Requesters ask for addition or removal of userrights (admin, checkuser, translate admin, etc) to their account on a given wiki. The process happens on Meta because it is the central coordination space for all wikis. Discussion takes place in English by default, but individuals can contribute in their own languages (participant reported using Google Translate to translate editors' comments in other languages). The process should be initiated on the home wiki, with a rationale for the request posted in a public forum like a Village Pump. On SRP, the discussion is initiated by the requester when they paste a filled-in request template on the section of the board that corresponds to the userright they are requesting. That template must contain a link to the previous discussion on their local wiki. Only Stewards can grant these requests, and make determinations. If the request is straightforward and noncontroversial, there may be no discussion at all. If the request is for a more powerful userright and/or if there is dispute about whether the requester should be granted it, discussion between the requester, Steward, and any other interested user may take place. There is no requirement that the requester or Steward perform any additional notifications before, during, or after the decision is made. Closed requests are archived by a bot.

Process stages

Initiating a proposal

Notifying people and projects. Notifying people that a discussion has been initiated is important. People who might need to be notified of a new discussion (potential stakeholders) include:

people who work on an article (FAC, FAR, AfD)
subject matter/domain experts (FAR, FAC, AfD)
closers/delegates/mediators/admins (FAR, RfC, Dispute Resolution, Deletion requests, ANEW)
people working on a draft (AfC)
image uploaders (Deletion requests)

Some proposal creation workflows include tools for posting a notification (on a userpage, article or project page, or noticeboard queue). But many community processes initial notification is largely done manually by the proposer. As a result, many people who would otherwise have relevant information to contribute to a discussion aren't aware these discussion are happening. People who are responsible for guiding or deciding a proposal (such as AfC delegates and dispute resolution mediators) rely on their watchlists, Special: feeds, page histories, and (in a few cases) alert bots to track new proposals that come in.

In some community processes, notifications are optional and are either a courtesy or a proactive strategy for recruiting participants to join the discussion. But other processes (ANEW, FAR) require notification of particular interested parties, but don't provide supporting tools for notification.

Creating the discussion thread. Proposing a new discussion can be a challenge even for experienced editors. Some processes (AfC, DRN, Deletion requests) provide structured workflows with good instructions that end with the creation of a properly-formatted discussion thread. Others (FAC, BAG, Requests for permissions, RfC) provide instructions on their portals which detail the steps for initiating a new proposal. In these cases, creating the thread usually involves copying and pasting a wiki template into the designated section of a page (either a page on the portal itself, or a content-namespace talkpage), or creating a new page with an inputbox and a preloaded template, and then filling in required parameters. It's easy to mess up.

Discussing a proposal

Gaining critical mass and maintaining momentum. Proposals often languish, or fail to get off the ground, due to lack of participation. Many processes (AfC, RfC, PR) have large backlogs, and it can be difficult to get enough people to participate to move the discussion forward, or to get a closer to resolve the discussion.

Tracking discussion activity. Editors use watchlists to monitor the progress of discussions. But as many editors have thousands of pages on their watchlists, it is easy to miss new comments on a discussion you're following without manually checking the discussion thread regularly. In portals that host all discussions on a single page (DRN, SRP, etc), you can check the edit comments on the page history to look for new comments.

Working in stages. Many processes operate in time-bound stages: a discussion cannot be closed until a specified time-interval has passed, or a discussion allows a particular time interval for reviews to be submitted and addressed, after which it is open to support/oppose voting. Getting an individual to return to the discussion (for example, to "sign off" on your changes after you've responded to their review) can be accomplished by manually @notifying them. Putting out an "open call" to get a whole group of people (such as all the discussion participants up to that point) to come back and vote is not currently supported.

Resolving a proposal

Finding a closer. In many community processes, closers are a scarce resource. It's not always clear who is allowed to close a discussion, or who is available to do so. In the case of FAC and FAR, only designated coordinators can make a determination of consensus and close a discussion. In DEL and ANRFC any admin can close the discussion. In some processes (DRN), a closer assigns themselves at a particular point in the discussion. In other processes (AfC) the criteria for becoming a closer are less stringent, or open to anyone who is willing to volunteer themselves and agree to the local rules (PR).

Making a final decision. For closers, coming to a decision, and justifying that decision, can be time-consuming. However this is a very important part of the process. Closers of discussions that involve long time scales, many participants, and/or lots of back and forth reported that their decision-making process was quite involved. They usually combed carefully through the various arguments, paying more attention to the strength of the arguments than the number or volume of the voices in support/opposition of the proposal. Since supporting material (policy citations, links to precedents, and other evidence) are buried in long threads and walls of text, this takes time. Participants also reported on the importance of crafting a succinct but informative rationale for their decision.

Documenting the results. The results of previous discussions can be used to inform subsequent decisions around related topics. On English Wikipedia, performs the task of retroactively sorting closed deletion discussions by topic area, as a resource for editors looking for precedents to support the deletion or retention of other articles in that topic area. Other processes, such as RFC, maintain an of previous discussions organized by month.

In Featured Article Review, the result of the discussion (keep, delist) and a link to the discussion are posted by a bot on the article talkpage ().

Common patterns

Several common patterns were observed across the community processes we investigated. Other features varied widely between processes.

Use of supporting tools

Many processes provide supporting tools such as bots, userscripts, gadgets, and templates to streamline parts of the workflow. Supporting tools help speed up repetitive actions to allow closers to manage a higher volume of proposals, facilitate the creation new proposals according to a specific template, deliver notifications, and archive completed discussions.

Examples

DRN proposal workflow
Commons QuickDelete gadget
AfD bot
AfC Helper script
Twinkle

Triage dashboards

Several processes also provide dashboards or indexes of open discussions, sorted by recency, topic, and/or status, in order to help closers decide which proposals to work on. The dashboards are either updated manually, or managed through transclusions of special pages (Special:RecentChangesLinked), extensions (DynamicPageList, and bots (Dispute resolution ClerkBot).

Examples

Variables

Location of the discussion
Meso-structure of discussions:
Scale of the discussion:
Rules and roles:

Recommendations for deployment

Pilot tests - start with smaller, simpl(er), less essential process. Get support of essential process participants. Ask them to experiment with configuring and using the CP in a sandbox wiki, provide clear mechanisms for feedback Arrange a time-limited pilot with clear feedback mechanisms and the ability to restore the legacy workflow if desired, at the pilot's conclusion. Survey/interview all participants (core and peripheral) at the conclusion

Resources

Wikimania 2015 presentation. Includes some findings and next steps.