성장 팀/질문방에 집중

그룹:	Growth
시작:	2018-10-01
팀원:	Stephane Bisson (software engineer), Benoît Evellin (community relations specialist), Elena Tonkovidova (QA engineer), Kosta Harlan (software engineer), Morten Warncke-Wang (data analyst), Rita Ho (senior user experience designer)
백로그:	#T206711
리드:	Roan Kattouw
관리:	Ryan Kaldari (engineering), Marshall Miller (product), Jazmin Tanner (program)
업데이트:	Most updates posted here.

This page is a translated version of the page Growth/Focus on help desk and the translation is 20% complete.

Outdated translations are marked like this.

파브리케이터에서 추적됨
일감 T206711

이 문서는 성장 팀의 "질문방에 집중" 프로젝트의 일부입니다. 이 문서에는 주요 요소, 디자인, 결정 사항을 나열하고 있습니다. 대부분의 순차적인 업데이트는 성장 팀 업데이트에 게시되며, 대규모 업데이트가 이곳에 기록됩니다. Most incremental updates on progress will be posted on the general Growth team updates page, with some large or detailed updates posted here.

이 프로젝트를 돕기 위해 제작 중인 "도움말 패널"의 [$prototype 동적 프로토타입]입니다. 이 문서에 이 기능에 대한 내용이 설명되어 있습니다. 테스트 위키에 새 계정을 만들고 환경 설정의 "편집" 문단에서 기능을 활성화한 후 아무 문서에서나 편집을 시도함으로써 이 기능을 사용해 볼 수 있습니다.

As of September 2019, we have finished analyzing experiments that show the help panel did not have impact on activation or retention of newcomers. Therefore, we'll not be iterating on that feature going forward, and instead focus on the promising newcomer homepage and newcomer tasks features. See details on the experiment results here.

현재 상태

2019-09-30: finished help panel experiments and published topline results.
2019-07-22: help panel deployed in Arabic Wikipedia in an A/B test, with half of newcomers not receiving it.
2019-03-26: 데이터 지표에 대한 최초 보고서를 게시하였습니다 .
2019-03-04: 도움말 패널이 도움말, 도움말토론, 위키백과, 위키백과토론, 사용자, 사용자토론 이름공간 문서에서 문서를 읽을 때 표시됩니다.
2019-02-28: 도움말 패널이 베트남어 위키백과에서 A/B 테스트와 함께 배포되었습니다. 새 사용자 중 절반은 패널을 보지 않습니다.
2019-02-14: 검색 기능이 체코어 위키백과와 한국어 위키백과의 실험 그룹에 배포되었습니다.
2019-01-11: 한국어 위키백과와 체코어 위키백과에 도움말 패널이 A/B 테스트와 함께 배포되었습니다. 해당 위키의 질문방을 통해 해당 기능을 이용해 질문된 내용을 볼 수 있습니다.
- 체코어 위키백과 질문방
- 한국어 위키백과 질문방

Next: we are working on an incremental features to encourage users to ask questions, to improve their search results, and to increase the likelihood that they read their help desk responses.

Experiment results (September 2019)

The help panel was first deployed to newcomers in January 2019, and as of September 2019, we have now finished analyzing data to determine its impact. This is a brief summary, and a more in-depth analysis is forthcoming.
In summary, the help panel has not shown an increase in activation (whether a user makes their first edit) or retention (whether a user returns to edit again). While this result is disappointing, our team has several other promising features that we continue to experiment with.
The help panel does cause slight changes in edit volume (the number of edits made), but the results are conflicting. It increases edits during a newcomer's first day, but decreases them over the following two weeks.
In general, we do not currently know why the help panel causes these results, and we have many theories that we will continue to think about. For example, while we hope that the help panel encourages newcomers to make their first edit by giving them information on how to do it, it is possible that it discourages some newcomers from editing because it shows them a lot of complex documentation, which may be boring or scary.
Although increases in activation and retention are our team's ultimate goal, we also observe that large numbers of newcomers open the help panel (20%) and interact with it once open (50%). This tells us there is demand for the sort of help that the help panel offers.
Taken together, these results are disappointing -- we were certainly hoping that the help panel had a positive impact on the metrics. While we have many ideas for how to improve the help panel, we have decided not to iterate and develop on the help panel for the foreseeable future, and instead to keep our attention on the newcomer homepage and newcomer tasks. Those projects have promising early results and reach many more newcomers than the help panel does. The help panel will also be used as part of the newcomer tasks project, and will remain as part of the newcomer experience that the Growth team is building. We may revisit it in more depth in the future.

요약

새 사용자는 일반적으로 기술적, 개념적, 문화적 어려움을 겪습니다. 특정한 질문이 있는 경우가 있을 수 있고, 이러한 질문에 대한 사람의 도움이 유용할 수 있습니다. 대부분의 위키에는 새 사용자가 질문을 할 질문방이 있으며, 이러한 공간은, 특히 영어 위키백과의 찻집과 같은 공간 (이 공간의 영향력에 관한 [$paper 다음 논문]을 참조하세요)이 유용한 것으로 밝혀졌습니다. 문제는 대부분의 사용자들이 이러한 질문방에 대해서 모르거나 찾지 못한다는 것입니다. 대부분의 새 사용자는 환영 틀에서 질문방에 대한 링크를 보는데도 말이죠. They have specific questions, and could benefit from a human editor answering their questions. Most wikis have a help desk where new editors can ask questions, and such venues have been proven to be effective, especially the Teahouse in English Wikipedia (see this paper for an analysis of impact). The problem is that most new editors don't know about or find the help desk -- even though they frequently receive "welcome templates" on their talk pages with links.

두 가지 가설을 세웠습니다.

새 사용자는 질문을 할 수 있는 편한 방법이 있다면 도움을 구할 것이다.
새 사용자는 도움을 구할 방법을 알고 있다면 편집을 더 많이 할 것이다.

따라서 "질문방에 집중" 프로젝트는 두 가지 요소를 포함합니다.

새 사용자를 도움말에 초대합니다. 네 가지 아이디어를 가지고 있었습니다. 새 사용자의 토론 문서에 글을 남기거나, 배너를 사용하거나, 이메일을 사용하거나, 편집 상황에 링크를 추가하는 식이었습니다. 팀에서는 새 사용자에게 편집 중에 질문방으로 가는 명시적이고 명확한 링크를 제공하여 새 사용자를 질문방으로 유도하기로 결정되었습니다. 다음 이유로 이러한 기능은 "도움말 패널"로 명명되었습니다.
- 성장 팀은 새 사용자 경험 연구를 통해 맥락적 도움을 제공하는 것이 중요하다는 것을 알고 있습니다. 이러한 링크는 현재 도움을 받을 방법이 명확하지 않은 장소에서 도움이 가장 필요한 순간에 도움의 손길을 제공합니다.
- "맥락적 질문 또는 채팅"에 대한 열린 질문을 갖고 있으며, 이러한 아이디어가 이러한 기능에 수요가 있는지 여부를 파악할 수 있도록 해 줄 것입니다.
새 사용자를 성공적으로 질문방으로 이끌어낼 수 있다면, 질문방 그 자체를 질답을 하기 더 좋은 장소로 만들 수 있을 것입니다. 이러한 자세에는 새 질문을 위로 올리거나, 질문에 답이 올라왔을 때 질문을 올린 사람에게 알림을 제공하는 것 등이 있을 수 있습니다. This might include listing new questions at the top of the page or notifying new editors when their questions have been answered.

성장 팀은 첫 번째 부분 (도움말 패널)을 2018년 2분기 (2018년 10월~12월)에 개발하였으며 2019년 1월 11일에 통제된 A/B 테스트의 일부로써 배포하였습니다. 이 실험은 2019년 7월에 완료됩니다. 결과가 준비되면, 추가적인 기능을 배포하거나, 추가적인 위키에 배포하거나, 2단계 작업 (도움말 그 자체에 대한 개선)을 시작할 수 있습니다. This experiment will take until July 2019 to fully complete. As results come in, we may add additional capabilities to the feature, deploy it in additional wikis, or decide to begin Part 2 (undertaking improvements to the help desk itself).

Comparative review

Our team's designer reviewed the way that other platforms (e.g. SurveyMonkey, American Express, Codecademy) offer help in-context to their users (see 작업 T206713 for details). We think there are best practices we can learn from other software, especially when we see the same patterns across many different types of software. Even as we incorporate ideas from other software, we will still make sure to preserve Wikipedia's unique values of openness, clarity, and transparency. The comparisons are shown in this slide deck, and the main takeaways are:

In-context help UIs

In-context help is usually presented as a sticky button or tab in the bottom right of the screen
Clicking on help will usually open up a sticky panel over the UI offering one or more help options
Users are often able to search and view top help topics within the in-context help panel
In-context help will also link users to the dedicated/primary help center/forums
General/site-wide Help is usually a primary navigation option as well, and may also enable access to chat/feedback within the UI
Indicators of privacy and security also often shown in the expanded help UI
Mobile (app/web) versions typically do not have help accessible in-context but accessed as a primary navigation item that opens separately

Types of help offered in-context

Link to a Help Center / Help Desk
Top FAQs listed for that particular context
Live chat
Video/Tutorials
Guided tour links
History of previous help queries
Community forums - used in a couple of places where there is less expectation of immediacy in response times and level of ‘service’
“Report a bug” - for users to send feedback about unexpected behaviors/bugs, often with the option to include a screenshot of the UI. Implicit expectation is the user should not to expect a response.

Features of Chat/Q&A in help

Users are able to email questions using the chat panel when there is no “Live” agent is available
Users are given feedback on response time windows and where they can expect answers
Voice and tone of chat are friendly and personable, sometimes with emoji and gif options in chat.
Users likely expect a level of service, as chat is typically provided by paid customer service reps.
Privacy links and information about how chat history is managed are clearly shown
Frequently asked questions/topics common for the particular context are often presented in the same UI as an less engagement-heavy help alternative for users
Automated answers / “Answer bots” are sometimes offered first to try to resolve the issue before users are given the option to post their question

이 아래의 문단은 근시일 내에 대폭 변경될 수 있거나, 이 프로젝트를 이해하는 데 별로 유용하지 않거나, 너무 기술적이라고 여겨지는 내용들입니다. 이 아래의 내용은 번역하지 않기로 결정되었습니다.

Review of similar MediaWiki features

As we discussed ideas for the help panel, several WMF staff reminded us that two similar-sounding features had been worked on in the past: MoodBar and Article Feedback. Those features (though no longer in use) both gave users an in-context tool for providing information, and so we think we can incorporate learnings from them. The details of this work are on 작업 T206714, and the main takeaways are:

Screenshot of Article Feedback Tool

Article Feedback Tool

Potential for account creation and editor activation - anonymous users who start an edit may be prompted to create an account when asking for help so that they can get feedback from the “Help team”.
Provide more upfront help options prior to allowing submission of free-text comments and questions - Per findings from the Article Feedback Tool, making it too easy to submit unstructured and out-of-context comments about articles leads to a high degree of noise for moderators. Whilst our hypothesis is that users who are in the editing context and seeking help for a specific task may be less likely to submit un-useful questions, providing more self-directed help alternatives upfront may further reduce volume.
Mechanisms to provide context to comments – The lack of context to comments in AFT was a main point of dissatisfaction for editors. For in-context help, it would be ideal if users submitting a question or comment would be able to denote where in the article or UI they would like assistance – for example being able to submit a screenshot along with their question, or being able to highlight the section related to their comment.
Making interactions into actual edits - This tool created a separate queue of work for experienced editors to deal with, that they couldn't manage through their Watchlists or RecentChanges, and current curate with existing processes for removing damaging content. It is important that any additional traffic we create can be managed through existing systems, which is done most easily if interactions with our feature are done as posts on wiki pages.

Screenshot of MoodBar

MoodBar

Capturing email for responses is an effective option – As shown in the MoodBar experiment, email notifications is an effective means of engaging new editors.
Clear, prominent call to action for new editors to ask for feedback/help - per high level findings, the addition of a tooltip drawing attention to the MoodBar significantly increased its use.
Tracking the perceived helpfulness of responses is important - It would be useful for us to track similar new user perceptions of feedback, since a poor experience from receiving unhelpful advice has been shown to negatively impact editing.

Design for help panel

Our evolving designs can always be found in this interactive prototype. These clickable mockups also contain other design ideas for future iterations of the help panel, although some elements in those mockups may be obsolete. To see the latest version of the help panel, edit a page in Beta or Test wiki after having turned the help panel on in your preferences.

The "Comparative review" and "Review of similar MediaWiki features" were critical in the design process because they helped us "dream big" to explore the space of what the help panel could eventually become. Our team mocked up and discussed many ideas that probably will not become part of the help panel unless we see that earlier, simpler versions are successful. First, we are going to work on an initial version of the help panel to be deployed in January 2019.

Initial version

In the initial version of the help panel, we want to answer these questions:

Will newcomers click on a clear option to get help during the editing experience?
Do newcomers seem more interested in reading content to answer their own questions, or in asking a question for others to answer?
Will newcomers take action to write and post a question to the help desk?
Will the presence of the help panel increase new editor retention?

Therefore, we have decided to include these elements in the initial version, detailed in T206717 and T209318:

A call to action in the editing context.
A panel that opens containing links to helpful existing pages.
The ability to ask a question from that panel, and for that question to be automatically posted to the wiki's help desk.
The ability to add an email address to their account if the user does not already have one, and to modify their email address if it is not yet confirmed (and then to receive a confirmation email upon submitting).
The ability to turn off the help panel by clicking through to Preferences.

The mockups below show initial designs for that functionality as of November 2018 (to see the most up-to-date designs, use this interactive prototype). Right now, we will show this only to newcomers (meaning new account holders who are not auto-created). We also know that the wording in the feature is very important here, because it is critical that newcomers understand where and how information they write will be posted, and when and where they can except a reply. These mockups only contain some initial drafting of the wording, and we'll continue to refine it. We are hoping for strong ideas from community members, so please post any ideas on the talk page.

1) Mockup of initial call-to-action on desktop and mobile
2) Mockup of help pane initial state on desktop and mobile
3) Mockup of asking a question in help pane
4) Mockup of confirmation received after asking a question
5) Mockup of a question having been automatically posted to a help desk

Business rules for initial version

Below are the current rules we're using to determine who receives the help panel and how it works. These may change as our team and the community continues to discuss and learn from the feature.

Users: the help panel will be a feature that can be turned on and off in user preferences. When it is deployed in January, it will be turned on to only newly created accounts (not auto-created) from the deployment date forward (except for a control group that will not have the feature). All other users whose accounts were created before the deployment date will have the feature turned off, but will be able to go to their preferences to turn it on if they wish to try it out. We will also be embedding an option to turn the help panel off inside the feature. Our team discussed the possibility of turning the help panel on for all users, but wanted to first learn about how new users interact with it, and we want to make sure not to flood help desks with incoming questions. See this Phabricator task for the discussion.
Namespaces: the help panel will be available in all namespaces. Our team discussed limiting the feature to only the main article namespace because the help documentation linked in the panel is mostly relevant to that namespace. Ultimately, we decided to make it available in all namespaces so that we can learn in which namespaces people are most commonly looking for help -- we know that new editors struggle to edit the Talk namespace. We will also encourage communities to develop documentation relevant to editing other namespaces (particularly the Talk namespace) so that we can make those links available in the appropriate namespaces. See this Phabricator task for the discussion.
Headers: the help panel will automatically post users' questions to their wiki's help desk, and will automatically generate a header for that post. We have decided that the header will include the title of the page from which the question originates (with a link), as well as a timestamp to distinguish it from questions that may have originated from that some page. Users will be able to decline to include the title in their header, if they consider it sensitive. See this Phabricator task for the discussion. Our preference is to number the posts instead of including a long and redundant timestamp, but that will require additional engineering work for a future version.

Wording for initial version

The wording for the help panel is complete, and the current state can be seen in the interactive mockups. Some of the most important things we have been considering in the wording are:

How to make it clear that by asking a question in the help panel, users will be posting in public?
How to encourage users to add their email address, since that is the best way for them to find out when they get a response to their question?
How to give users the option of excluding the title of the page they're editing from the automatically-generated header in the help desk, if they want to keep that private?
How to correctly set expectations about how quickly a user can expect a help desk response?
What to include in the automatically-generated header in the help desk?

Future versions

In determining the initial version of the help panel, many ideas for features were set aside to be evaluated later on. Many of them are visible in the initial mockups here. The following is a list of capabilities that are either being built for the help panel, or may be built in the future:

Nearer term
- - Search: the ability to search help pages. Users found this option appealing during testing, and it was built in T209301 and deployed on 2019-02-14.
  
  Wikipedia help panel feature in Korean Wikipedia, with the "search" functionality in use.
- - More contexts: since help desks have not yet been overwhelmed with incoming questions, we are thinking of exposing the help panel to newcomers in contexts beyond editing -- perhaps while they are viewing pages in the Help or Wikipedia namespaces. We thought of this because the EditorJourney analysis showed that large portions of newcomers viewed pages in those namespaces during their first 24 hours. Currently being built in T215664.
- Greater affordance: so far, it looks like a minority of users who see the help panel button actually open it up. We could attempt to make it more visually obvious that it is a recommended place to start.
- - Vary links with context: the help panel contains five links to help pages, and about half of users that see them click on one. Different links are probably valuable in different contexts. For instance, there might be some links for visual editor and some for wikitext. Or some that are helpful on a talk page and some on an article page. This is being worked on in T211117.
Longer term
- Live chat: allowing newcomers to immediately chat with experienced editors right from the help panel. This would be a major project with many challenges. See this page for our notes so far.
- Suggested answers: many questions that users asked may have already been answered before, and we might be able to provide suggested answers as users type. This would be relevant once many more questions have been asked. Ticketed in T209327.
- Feedback on responses: newcomers may find the responses to their questions more or less satisfactory. By allowing newcomers to rate their responses, we may increase higher quality responses, and make it possible to surface the best ones to future newcomers. Ticketed in T209332.
- Marking question context: newcomers may have questions related to very specific elements of the editing experience. This idea would allow them to indicate exactly where on the page their issue is occurring. Ticketed in T209328.
- Templated replies: research shows us that the tone of a reply to a newcomer can make a difference in whether they edit again. We may be able to help people answering questions to format their replies in a nurturing way. Ticketed in T209331.
- Anonymous editors: though our team is focused on registered newcomers, it is possible to make the help panel present for anonymous editors, though it will be more difficult for them to find out that they have received a response. Ticketed in T209325.

User testing for help panel

During the week of November 26, 2018, we used usertesting.com to conduct eight tests of the help panel interactive prototype with internet users unaffiliated with the Wikimedia movement. Four tests were done on desktop and four were done on mobile. In these tests, respondents are compensated for trying out the mockups, speaking aloud on what they observe, and answering questions about the experience. As our team's designer described on the Phabricator task, the goals of this testing were:

Gauge the discoverability of the help pane call to action and help pane
Identify improvements to the usability of the help pane:
- Do users understand they can edit and use help links at the same time?
- Are users able to successfully follow the steps to post a question?
- Is there anything users that would like to be able to do from the help pane that they are missing?
Gauge user reactions to the help pane and expectations of how their questions will be answered.

Summary of findings

All users found the help panel contents to be very useful in getting help for editing.
No one had issues with having the article title included in the question.
Most participants would post a question to the help desk as a last resort after trying to help themselves through clicking on links, with two people saying they would likely not post at all as they would want a more immediate response.
One user said they would more likely just Google the editing help over the options shown.
Whilst it was understood that the Help desk would be answered by volunteers, many participants wanted more reiteration and clarity of expected wait times.
Clear layout of options, shown in the order in which almost participants said they would seek for help.
Discoverability of the help panel call-to-action may be reduced in the case when the user is using the Visual Editor or wikitext2017 editor because those editors have a competing “?” icon.
One user suggested having an example question in hint text.
One user wanted more information shown about when their question was viewed and by how many people.
Two users mentioned the help recommendations shown on the Help Desk seemed disjointed from what was shown in the help panel.

Recommendations (not all of these will be implemented)

Prioritize the Searching help task inside the help panel.
Provide clearer times on both the review and confirmation screens for when a newcomer should expect a response from the help desk.
Add a more specific help question sample in the placeholder/hint text.
Experiment or A/B test alternative text on the initial “Ask a question” call-to-action.
Show a first-run tooltip/pulse indicator to highlight the new help panel.
Add an additional menu item to existing editing toolbar help on Desktop VE or Wikitext2017 which opens the help panel.

Measurement and results

Measurement and experiment plans

High-level questions

These are the main things we want to find out from the help panel.

Does the presence of the help panel lead to questions being asked and answered?
Does the presence of the help panel increase editor activation (making their first edit)?
Does the presence of the help panel increase editor retention (coming back to edit again)?
Does the presence of the help panel increase the proportion of constructive edits?

Controlled experiments

See this page for the detailed experiment plan.

In order to understand the help panel’s impact on editor activation and retention, we propose a six month A/B test. During that test, 50% of new registrations on target wikis will have the help panel enabled by default, and 50% will have it disabled. We anticipate that we will be able to detect 10% changes in activation after about one month, and 10% changes in retention after about six months. We are likely to be running other experiments on the target wikis at the same time, for example, testing variants of our welcome survey. Therefore, assignment of users to treatment and control groups for all tests will have to be coordinated so that we ensure the distributions are not biased.

Specific measurements: quantitative

These are specific measurements that help us answer the high-level questions. These are measurements we will be able to do programmatically. See this EventLogging schema for the exact details on the data that is being recorded and stored.

What is the context the user is in when they start interacting with the help panel?
What percent of users open the help panel?
What percent close the help panel without using any part of it?
What percent click one or more of the help links?
How far do users go in the workflow of asking a question?
What percent of users turn the help panel off?
What is the average length of questions asked via the help panel?
What percent of users click any of the links in the final confirmation page of asking a question?
What percent of users who post a question return to view the answer?
Does interacting with the help panel alter the probability of abandoning an edit session?
What percent of users use the help panel more than once?
Are newly registered users with an email address likely to return to view the answer to their question?
What percent of newcomers that had no email address add one when asking a question?
What percent of newcomers confirm their email address (within 48 hours) after asking a question?
What percent of newcomers ask a question without an email address?
What percent of newcomers submit a question without including the title of the page they were editing?
How does help panel usage vary with welcome survey responses?

Specific measurements: qualitative

These are specific measurements that help us answer the high-level questions. These are measurements we will need to do manually, by counting and reading the contents of help desks. Doing these programmatically would be an inordinate amount of work.

What percent of questions get answered?
How quickly do questions get answered?
How often do newcomers edit the question they asked?
What kind of questions do people ask and which ones are more likely to get answered?

Usage counts

After about two months of being deployed to half of new accounts in Czech and Korean Wikipedias, and two weeks of being deployed to half of new accounts in Vietnamese Wikipedia, these are the numbers (as of 2019-03-28):

3,343 users have seen the help panel button
566 users have opened up the panel
225 users have clicked a link in the panel
51 have run searches
51 users have submitted a question to the help desk (Czech: 34, Korean: 8, Vietnamese: 9)

Leading indicators

The help panel's experiment plan defines a set of leading indicators that the team evaluates in order to determine whether the feature is behaving as expected, and whether there are any urgent problems that need to be solved.

Using a month of data, we published an initial evaluation of leading indicators here. In summary, the help panel seems to be behaving in a healthy way, with good numbers of newcomers opening and using it. The analysis identified a couple areas for improvement that the team has taken action on:

Because we wanted to increase the number of newcomers who open the panel, we started showing it in additional contexts on March 4, 2019. Newcomers now see the help panel when reading in the Wikipedia, Help, and User namespaces.
Because we saw that in Korean Wikipedia very few users ask questions, we built the search feature so that newcomers would have an easier way to find helpful information on their own.

Experiment results

See results above.

Resources

More on this feature: The help panel contains a link to learn "More about this feature". That link will lead to this page, which will hopefully answer user questions about how the feature works. Feel free to translate it into your language!
Best practices: Because many communities don't have a lot of experience concerning help desks, the WMF Community Relations Team has assembled on a page with some best practices that can help communities have more successful collaborations with newcomers. Please feel free to translate that page into your language!