Talk:Growth/Focus on help desk

It increases edits during a newcomer's first day, but decreases them over the following two weeks.

I support improvements, exploration and experimentation. A project may of course require a few revisions before reaching positive results.

If the team doesn't find a way to reverse the longer term negative impact, can someone confirm that there is an explicit plan for full rollback? This is a general concern to avoid accumulating complexity, cruft, and inconsistency across our wikis when some projects don't work out.

@Alsee -- thanks for following along and reading our notes closely. The help panel indeed has ambiguous results. On the one hand, we see anecdotal stories of users who ask questions, get answers, and then continue to edit. On the other hand, we don't see an impact on activation and retention, and we see this ambiguous impact on edit volume. So one thing that you're pointing out that we don't know is whether the help panel seems to increase or decrease edits over the entire first two weeks. I think that so far our analysis looks at Day 1, finds a positive impact, then looks at Days 2 - 14 and finds a negative impact. We can look into what the overall impact is.

But beyond that clarification, our current thinking is that the help panel may work better as part of a broader "integrated newcomer experience", along with the welcome survey, homepage, and newcomer tasks. In other words, it may not be a specific feature that makes the difference for newcomers, but a general overall upgrading of their experience via several connected features. In fact, for newcomer tasks, we plan (in the January/February range) to evolve the help panel to be the feature that guides newcomers as they complete tasks.

All this is to say that we want to keep the help panel around for now and integrate it with the newcomer tasks experience, and see how that affects its usage. If it is not proving useful in that context, then I think we'll need to revisit the questions you're bringing up -- because I agree that we should not leave up features that are not working.

How does this sound?

Thanx. If you reach more positive results that's of course a good thing, and I apologize if it seemed I was jumping the gun about the initial results. I appreciate your confirmation that we should not leave up features if it turns out they're not working. To be honest, the topic caught my interest because there have been difficult times where teams have resisted their own data when it was unfavorable or counter-intuitive. It's easy for good intentions and big positive goals to feed into confirmation bias. The most crucial metric for any project is our inflow of new users as sustained-activity community members. Any negative data in that area should be taken very seriously.

"we also observe that large numbers of newcomers open the help panel (20%) and interact with it once open (50%)" -- how can the first percentage smaller than the second?

@Samat -- thanks for reading closely. It's because the 50% is 50% of the 20%. In other words, of the people who open the help panel, half of them interact with it. To get the overall percentage, you multiply them together (i.e. 10% of people who see the help panel button interact with something in the panel). Does this make sense?

Thank you, that's clear for me now. (And I made it clear in the translation as well.)

The tips on how to post a good question currently says:

• If you know that information, mention whether you are editing with the visual editor (you directly edit and format the a article) or with wikitext code (you edit the code that builds the article).

That is poorly worded, confusing and arguably factually erroneous. I suggest one of these two versions:

• If you know, mention whether you are editing with the visual editor (graphical editing mode) or with wikitext (text editing mode).
• If you know, mention whether you are editing with the visual editor (graphical editing mode, links look like this) or with wikitext (text editing mode, links look like [[this]]).

Thanks for the suggestions, @Alsee. I'm tagging @Trizek (WMF), since he is managing that page.

Thank you for the suggestion. I've used the first one and added some links.

Note that I changed "wikitext code" back to "wikitext". That phrasing is not normal usage outside of Foundation, and would be an inconsistent and poor choice for public consumption.

A search for wikitext on EnWiki gives 117,499 hits, while a search for "wikitext code" gives 329 hits (a ratio less than 0.003). Most of those 329 hits use the phrase to refer to a specific string of wikitext, many search hits were written by a staff member or were incidental in bulk copypaste of text written by a staff member, some are false-search-hits, some are in a technical context contrasting wikitext from lua or other programming language.

Referring to wikitext as wikitext code really is virtually non-existent English usage outside of the Foundation.

This is a topic for community discussion around the rules, wording, and design of the help panel.

In User testing for help panel, the comment "Discoverability of the help panel call-to-action may be reduced in the case when the user is using the Visual Editor or wikitext2017 editor because those editors have a competing '?' icon", is certainly an issue. In VE [I'm less certain about the wikitext2017 editor], the dialog invoked by clicking on that "?" icon should be changed. It now begins "If you encounter any technical issues as you edit, please report them," which (a) is not likely to be why the user clicked on the icon, (b) is duplicated by "Leave feedback about this software", and (c) made sense back in the day when this was beta software, but it's not anymore.

As to what the VE dialog should look like, I suggest "Help:" as a heading, followed immediately by four items: "Open the help panel", "Read the user guide", "Keyboard shortcuts", and "Provide feedback on this software". That (obviously) gets the user to the help panel in just two clicks, while offering some other options.

Also, to quibble, a help icon is usually in the form of a question mark with a circle around it, not a naked question mark. And the icon for the user guide (and maybe the help panel), within the dialog, should be an "i" with a circle around it, for information.

Thanks for thinking about this, @John Broughton. We also were thinking about the idea of letting the help panel be an option in the "?" menu. But what do you think about the help panel potentially just superseding the "?" entirely? And removing it from the toolbar? We are going to be able to keep an eye on this, because we recently started tracking how often users open the "?" menu.

The only thing that I think is useful in the current ? dialog is the link to the user manual; I'd hate to lose that. Maybe split the difference - have a "circled i" (information) icon that links to the user manual, and a (circled) ? that offers help?

Alternatively, something that has been discussed at some length, but not implemented (for which I'm probably the most to blame, since WMF resources were offered to me a while back, to test the concept) would be to put links to various sections of the user manual in context-specific places (specifically, the various pull-down menus, and menu items of VE). That would obviate the need to put a main link somewhere.

Yeah, I was thinking that we could put the link to the user manual in the help panel, so that we would consolidate all the helpful links in one place. How does that sound?

The idea you described about put links to various sections in context-specific places sounds like the "In-context lessons" idea that you helped us discuss a few months ago. I'm hoping that after we get some learning through the help panel, we might turn attention to that idea.

Sure, a link in the help panel, to the user manual, is quite satisfactory.

Also, I'm guessing (I don't see an explicit statement on this) that you're relying on a volunteer editor's response to a user's question to actually include the user name, which will then trigger an email to that user [I think]. However, while including the user name in a response is common, it doesn't seem universal - see, for example, https://en.wikipedia.org/wiki/Wikipedia:Help_desk#Fixing_Odd_Formatting_on_Article as well as the question and answer immediately below that section. And it's not mentioned in best practices at Growth/Communities/How to work with newcomers on help desks . Are you planning to automate/ensure that an email is in fact generated for the new user when the user's question is posted?

Concerning the point about mentioning people, it is already in the best practices: "Make sure the newcomer is aware of the reply, by any method."

Thank you for pointing it out, though: I've clarified the sentence by having it as a separate point and added "For instance, on wikitext talk pages, mention the user when you reply."

@John Broughton -- an idea that @Martin Urbanec brought up is to use an abuse filter to remind responders to mention someone in their reply. Do you think that could be a helpful approach?

While using the abuse filter has potential, I worry about the details. Specifically, not all edits require a user name - the first in a section, obviously, but possibly the second, because sometimes the user with the questions posts and then edits his/her post, or adds a second post immediately. Or the pattern might be (1) question, (2) response, and (3) elaboration on the response by another (helping) editor; if so, I suspect that the editor of the third post would not appreciate getting a warning, since the second post should already be notifying the user who had the questions.

Regarding "Provide clearer times on both the review and confirmation screens for when a newcomer should expect a response from the help desk", that is obviously going to vary by community, and there isn't any way to pick a single number (or range) that is both accurate and helpful across all communities. (For example, "one day to two weeks" is probably accurate, but not helpful; "Within 24 hours" is helpful, and for the English Help desk is accurate, but may well not be for many other communities.

So, two options come to mind - either each community can specify that parameter (hidden comment on the Help desk page? central list that is updated by stewards upon request from an editor?), or an automated way of calculating the average time - say, a bot that checks once per day to compare the date/time stamps in each of the most recent ten sections that have both an initial posting and a posting that is [presumably] in response [different user name].

@John Broughton -- that's a good idea about letting each wiki specify their usual response time! We're going to hang on to that idea. For now, we know that Czech and Korean Wikipedias (the pilot wikis for this project) can commit to the "day or so" response time, and so we have it hard-coded that way. I hope that if this feature works, and if communities develop enthusiasm around it, they might be able to commit to shorter and more specific response times.

That response time may change over the year, since some people may be away, or some events may slow down he activity on wikis.

One way to deal with a response time promise that hasn't been kept is to provide a link/option for the new user to report that "I have not had a reply within the promised time". That report might go to OTRS, for example, or to someone specifically charged with maintaining the response table.

Interesting idea. It sounds like something we could include when we think about a way for users to give feedback on their experience with the help panel. We were thinking about that here on Phabricator, and I've added a note.

Some ideas are described here: https://www.mediawiki.org/w/index.php?title=Topic:Uq5p24kedymiw6pq&action=history.

I'd say the greatest failing of the current design is the fact that it includes an ability to immediately ask a question before even searching, and it makes it optional to add a page title. Neither should be visible in the first page, and it should not be possible to choose to include the page title as context, as that should always be included in the feedback.

In the short term it might be reasonable to include an extra item in the drop down for visualeditor, as suggested above. Long term, they should be combined into a button that routes the feedback to the appropriate place, either phabricator (for technical matters), VisualEditor/Feedback (for general feedback), a help desk (for general support), or a talk page (for page specific issues) depending on the nature of the problem or feedback.

Also, eventually there should probably be a non-javascript fallback, maybe Special:Help for when either editing without javascript or for users with temporary network glitches.

I think a non-JS fallback is essential; anyone editing on a low-bandwidth or expensive connection has a strong motivation to not use JS. Things that require JS, or lots of bandwidth, risk increasing systemic bias.

Thanks, @HLHJ. That's an important point. Right now, for non-JS users, the help panel icon shows up just like for other users. When they click it, however, instead of the dynamic experience of interacting with the help panel from the editing context, it opens up the help desk itself in a new tab. At that point, the user would be able to follow the directions on that page to ask their question. Do you think that is a good fallback? Can you think of other ways to do it?

Issue

Sometimes the problem lies with the tool, and the editor might not even know what they are using.

Proposed solution:

Always include the Editing tool being used in every single feedback post / thread.

Thanks -- this sounds like a good idea that we'll keep in mind.

Issue:

From the mockups there doesn't seem to be any way to search a question archive.

Proposed solution

Build a way for users to search any "approved" question.

Note:

This may need to take into account the fact that people may put non-sensical gibberish into the search box. So it might be preferable to only add moderated questions in the search index.

Thanks for recording this as a specific topic -- we actually have a Phabricator task for it! Here it is: https://phabricator.wikimedia.org/T209327. I made a note on the task that you brought this up.

Design issue:

The design emphasizes asking questions rather than retrieving existing ones. It also doesn't seem to facilitate storage of common questions.

Background:

The greatest flaw of the wiki concept is inefficiency. Consider this, what is the question always asked by users? How many times has it been asked? How many person-hours and resources and money has been wasted answering the same question over and over again over the 15+ year wikimedia projects have existed? Finally, how many times has it been asked in other language projects? The answer to all those questions is that nobody knows. As it stands this system will make things worse by making it too simple to ask questions without verifying first.

Concrete issues:

• The design has a checkbox asking to include a title.
• There are three basic types of questions asked, each needs its own flow:
  • Technical questions - without platform details.
  • Trivia questions - "Why is there a naked picture???", "Is blue pokemon tougher than green?", "Why can't I add my blog?". A good number of these are worthless, the rest should probably be directed to the article talk page instead of a help desk.
  • Topic related questions - These questions may be useful and may help improve the page.
  • Spam question - someone just writing random stuff there

Possible Solutions

1. Never allow the user to ask a question until they search first.
2. Never allow a question without a topic.
3. Always include page title as always an important context, oldid revision is also important.
4. Most importantly, it would be preferable to have a way for end-users to promote certain questions to the help pane. Maybe each newbie reply would come with a link that enables users to save the question with a standard answer. This would increase the pool of questions / answers, and overall reduce the eventual work.

Improving questions

• Technical questions
  • These must always (not optionally) require platform details (OS, Browser, etc). It is a waste of time to ask users these questions as follow-up.
  • It should also never be simply unstructured because it will result in a lot of unactionable or invalid reports. See VisualEditor/Feedback for a lot of messy questions, see also T86956
  • End-users often break their own interfaces using badly designed gadgets/ userscripts and browser extensions. First advise them to try to access the page using safemode, for example, https://www.mediawiki.org/wiki/pagetitle?action=edit&safemode=1, and possibly temporarily disable the extensions.
• Trivia questions - should be discouraged, and the interface should attempt to strongly indicate these are not the point before the user posts a question.
• Topic related questions - may need to go to the talk page instead of a help desk
• Spam question - Add some basic anti-vandal measures, e.g. don't allow single repeated character, e.g. "AAAAAAAAAAAAAAAAAAAAAAAAAA", have a sensible input length limit, maybe add captcha when questions include a link.

https://phabricator.wikimedia.org/T100011

Thanks for thinking about this -- it sounds like you have a lot of experience with questions from newcomers. I think you're right that there are several things we could do to help structure and store questions, and therefore improve efficiency in the future. And in fact, most of our user testers said they would try to help themselves by looking at existing help pages before asking their own question.

There are a few things about our approach and our learnings that have made us

We plan to release the help panel in Czech and Korean Wikipedias in January, and we plan to iterate on it over the course of at least six months (potentially adding additional wikis along the way). We're going to keep close watch on what kinds of questions are asked and how productive the help panel seems, and we'll be able to make adjustments as we go along. If questions contain trivia, spam, or topic questions, we can change the workflow to encourage other types of questions. But we don't want to spend too much time engineering that system at the outset before collecting some questions to see how they turn out first.

Another thing we're thinking about is the secondary benefits of newcomers asking questions. The primary benefit may be that the newcomer gets their question answered and figures out how to proceed. But our research has also taught us that newcomers are more likely to stick around when they make positive contact with other editors, when they see that there is a vibrant community behind the project, and when they feel like they belong. Because of those things, we're not sure if we want to discourage asking questions in favor of reading answers to existing questions -- we're figuring out how to balance that. It might become a different story if wikis end up overwhelmed with too many questions. Does that make sense?

And with respect to including page title -- we wanted to try making this optional (but checked "on" by default) in case it turns out that there are users who don't want to disclose the article they're planning on editing. We anticipate that most people will leave it checked. If we find that almost no one unchecks the box, we may revisit whether it's important to have it. If we find that many people uncheck the box, we'll want to look into why.

> I think you're right that there are several things we could do to help structure and store questions, and therefore improve efficiency in the future.

Seems to be something worth doing in advance, to save end-users and the developers a lot of unnecessary headache. Consider the difference between these two questions:

How do I get a reference to work?

vs

Question: How do I add a new citation / reference to a page?
Attempted steps: 
1. Open 2010 wikitext editor 
2. Clicked on the reference icon in the toolbar
2. The editing tool added <ref></ref>
3. Waited for a dialog
Expected
A dialog or anything to add a link or book or resource appeared
Actual:
No prompt to add anything, no help .

If "Question", "Attempted steps", "expected" and "actual" are different form fields, then question 2 is much useful than just a generic question 1.

> If questions contain trivia, spam, or topic questions, we can change the workflow to encourage other types of questions. But we don't want to spend too much time engineering that system at the outset before collecting some questions to see how they turn out first.

..

>Does that make sense?

This seems like a risky proposition. You may increase the burden on users, and then if the burden is too heavy you'll fix it? The fact of the matter is that the project deals with two variables it can't control, volunteers and development time. If they find it annoying they can simply ignore the questions, or answer badly and chase away newbies, or simply add automated tools to block such questions.

The project can be interrupted for any random reason, and it will leave users with a very limited basic tool. As an example, the tool being used here (Structured Discussions) was also an optimistic endeavor, yet it was mostly indefinitely halted in live sites, without even a simple ability to search existing threads, forcing users to have to rely on external search engines to find stuff here.

A similar endeavor exists in the English wikipedia's project teahouse with a question archive that allegedly doesn't even update anymore, and no sensible ability to search (https://en.wikipedia.org/wiki/Wikipedia:Teahouse/Questions/Archive_Index). So there's a lot of history of limited tools being abandoned in place.

> in case it turns out that there are users who don't want to disclose the article they're planning on editing

That's a legitimate concern only if users haven't ever edited the page. If they have already edited it, then the fact that they've interacted with the page will be known anyway. Perhaps, a compromise would be to only show the checkbox if they've never edited the page in question.

Thanks for pointing out some valuable things to keep in mind. I hadn't seen that Teahouse archive link. As we roll this out, we're going to keep close watch on how much of a burden we're putting on the experienced editors by counting how many questions are answered and how quickly. We're also in touch on a weekly basis with ambassadors in the Czech and Korean communities, @Martin Urbanec and @Revi (WMF), who are preparing their communities for these incoming questions, and who will be able to tell us whether things are getting too burdensome.

Although we originally started referring to the feature we are building in 2018 Q2 as the "help pane", we received some feedback that the name would be clearer if we called it "help panel". "Pane" sounds too much like "pain", and could be confusing. I updated the name on the project page and on the related Phabricator tasks.