Talk:New Editor Experiences/Flow

just curious...

Thank you! It's on this site mainly because we wanted people with less editing experience to be able to comment using Flow and also because the long-term goal was to figure out what software the Wikimedia Foundation could build to better support new editors.

Even to me, Meta feels like a more proper place, but in my experience content is divided between Meta, MediaWiki, and Wikitech so inconsistently that I've given up worrying about it.

>we wanted people with less editing experience to be able to comment using Flow

If you want to lower the barrier for responses from non-editors, you're better off dumping Flow and pointing them to a normal off-the-shelf internet chat board.

If you want to lower the barrier for responses from editors (even low-experience editors) you're better off dumping Flow and giving us a normal Talk page.

I don't want to make a wall-of-text trying to explain, but in short, the community rejected Flow, and Mediawiki is the only significant deployment of Flow, for good reasons. Flow is an obstacle to getting feedback or engagement from editors on Mediawiki.

Thank you. I am well aware of Flow's current status and your views on it.

This page was created more than two years ago and the project is long over (with the WMF's Growth team now making use of its recommendations). I need to update the page to reflect that.

Wikimedia Research is also on Mediawiki.

You can follow the next steps Neil mentions by visiting the projects aiming to help new users : Growth.

There are any number of ways to classify and think about the ideas we're evaluating—and, more importantly, to organize the recommendations that will come out of the New Editors process. But I'd like to argue for a way that really hit me as we worked through the user journeys exercise in our last session.

As I read the various user stories, similar themes emerged and coalesced in my mind. In particular, people had often faced similar challenges—and been helped in similar ways. With this in mind, the approach that will be most useful for our purposes, it now seems to me, is not just to evaluate which ideas are the strongest, or to parcel out different solutions that address different user personas, or to think about which assignments might fall into the wheelhouses of which WMF teams... Instead, I think we need to identify the key junctures in the New Editor journey, and then devise solutions aimed at addressing the users' needs at each phase.

Here are some examples of what I mean, just as straw men to illustrate what this might look like:

Juncture #1: Registration

• Ideas around onboarding, gathering profile information, welcoming, encouraging with recognition...

Juncture #2: exploring and finding a niche

• Ideas around progressive task suggestion, building awareness of the community, steering to Wikiprojects...

Juncture #3: first conflict/rejection

• Ideas around peer-to-peer assistance, improved policy pages...

The danger of not thinking in this way is that we run the risk of creating interventions that move users down the retention road a mile, only to drop them straight into the next (predictable) pitfall.

I'd always known that we wouldn't fix the problem of editor retention with just one intervention. But thinking about the new editor journey provides, I think, a useful framework for picking the most promising interventions and imagining how various efforts might fit together into a larger program of mutually-reinforcing parts. It's a way to turn a market-basket of ideas into a story about about what our users need and how we plan to help them. And, finally, asking which ideas help users most at particular points gives us more concrete criteria for evaluating and comparing very disparate ideas.

Re: Junctures: Yes. - I'm sure I've seen an infographic/project that demonstrated this idea quite well, but I cannot currently find it on mediawiki/metawiki.

The things that I found whilst looking, which are close but not exact matches, include (these are just the highlights, and should be semi-glanceable!):

• File:E2_Planning_-_Q3_2013.pdf&page=14 - Engagement Funnel
• m:Editor Growth and Contribution Program/Taghreedat#Workflow
• m:Editor Growth and Contribution Program/Contribution portal - Phase 0#Tutorial Pages
• m:Editor Growth and Contribution Program/Geo-targeted Editors Participation#Framework
• Growth/Prioritization
• Growth/2013-14 Goals#Our product framework
• File:Direct-to-page_onboarding_workflow_overview.pdf&page=2
• m:Research:Wikipedia Workflows/Article#Diagram
• m:File:Multimedia-Map-Workflows-August-08-2013.png
• Article Creation Workflow#/media/File:Article Creation Workflow diagram.png

Other related links I came across (whilst looking through search results for "funnel" and similar), as examples of some good prior work:

• Juncture #1: Registration
  • Editor campaigns/Technical design#Use cases and stories
  • Account Creation Improvement Project#Current account creation flow
  • outreach:Account Creation Improvement Project/Assumptions
  • Wikimedia Foundation Design/Login
  • Anonymous editor acquisition#Current user experience
  • m:Research:Account creation UX
  • Anonymous editor acquisition/Signup invites#Option 1: Pre-edit CTA

• Juncture #2: exploring and finding a niche
  • Onboarding new Wikipedians#User experience

• Juncture #h: Help gadget
  • New user help gadget

• Juncture #n: Post-first-edit
  • Post-edit feedback
  • m:Template A/B testing - fine-tuning the talkpage-warning template-messsages
  • m:Research:New editors' first session and retention#Summary
  • m:Research:Improve your edit

Lastly, this is still the best 10,000' overview (imo), albeit from 2011:

• Feature map

Hey Joe, FWIW, I agree with you 100%. It's different kinds of users at different junctures needing different things. Focusing on one at a time seems like an interesting approach, and i think focusing on specific user types and just a few junctions will serve us well. But we should be very careful, as you said, about what we choose to avoid opening up one door to help a user to find the next pitfall. The other thing that's been on my mind recently is how this isn't a funnel, but an ecosystem with many different components. If user type B moderates user type A, you can't 10x A without first 10Xing B's capacity (either through recruitment or better tools).

I agree on opening a door and then fall into an unexpected pitfall. From what I know, both from reading the interviews and personal experience, the big pitfall is the lack of human interaction: find someone to help, assist, discuss with, review...

Also, some recommandations we will publish can be handled directly by communities.

I agree these are relevant stages, and they seem useful to organise and evaluate ideas around them. I think that the stages identified are right, but I'd try to frame them in a way that is less coupled with technical solutions (such as registration) and have more clear connection with user goals (e.g., is the ultimate goal for a user to just "explore" or does such exploration has a purpose?).

I tried to adjust the language for the stages to reflect the above ideas:

• Feel invited to contribute. Covering registration process but also how we surface editors that an article is created by people like them and making them interested in joining.
• Find where to contribute in the best way. Surfacing interesting ways to contribute that are suited for their expertise, and new editors feel ready to accomplish (knowing not only how to use the tools to do it, but the key policy aspects that make a good contribution).
• Avoid issues and recover from them. Getting help and guidance from the system and/or humans when they experience a conflict or rejection of their work.

I agree also in being aware that the different stages are connected (you don't want to massively increase acquisition if you have big problems to retain those users). However, I also think we should not aim for the next stage to be perfect before doing any single improvement to the previous one, since that can also add unnecessary blockers. For example, if we have a simple idea to improve acquisition, I'd prefer trying it and deal with the potential case of becoming too successful (e.g., exposing it to a lower number of users) rather than postponing it in advance because the rest of the system is not ready for a potential huge influx of users (and not learning that the idea works so well for that stage).

It's a great idea to model the 'new editor journey'. Hopefully resources that help new contributors at each step can be 'plugged in' to this model. I suggest brainstorming (on-line) with some new editors.

Based on my newby experience, new contributors (including editors) don't go through the same learning process (journey). They certainly don't go through the steps in the learning process in the same order. Each new contributor follows his/her individual learning path (based on curiosity, immediate learning needs, the links he/she comes across, the editors he/she interacts with, etc.). The path that a translator or copy-editor takes will both be different to someone who joins to create a new article.

As an aside: the current WP learning materials/guidelines are IHMO not very suitable. They consist of a vast collection of long, text-based WP articles. For example, there are 367 "How to ..." articles". Many articles/guidelines are difficult to read, digest, remember and find again. Navigating between related pages (links, categories) can only be done by scrolling down to the bottom of each page.

One of the relevant opportunities in the REBOOT report was to "provide just-in-time guidance to new editors based on the tasks they want to accomplish". I think this would be a big improvement. There are very few Wikiproject training materials on YouTube and all pre-date the visual editor. I think there are opportunities to add training videos and on-screen tips (video/text).

In parallel with developing a model that includes all individual 'new editor/contributor journeys', it's worth considering how learning resources can best be organised to support any specific journey. E-learning (content) managements en e-learning environments may provide part of the solution for training materials. If these are linked by an API to the editors worksapce (in context), so much the better. On-screen tips are probably a separate issue.

Hope this helps the discussion, Mike

A whole set of interventions we've discussed revolve around the idea that encouraging users and showing them the impact of their edits will help them to be more active and successful. I personally like these ideas—I get excited when I get messages from Yelp, for example, telling me that hundreds of people saw my review.

But I just stumbled on some research around a very similar initiative. The experimenters set out to discover whether providing positive messages such as "25 edits. You're doing great!" lead the recipients to greater heights of editing activity.

The question gets asked and analyzed systematically in a variety of ways. The answer, across the board, seems to be that there is no measurable result. What do you think? Are you convinced by the findings? Or do you think the experiment was flawed?

I can see two main issues here:

1. In contrast to your example where people have read your review, there is zero human interaction in "25 edits, you're doing great". Don't get me wrong. I like to get these messages. But in contrast to your example where you literally reach more readers, these "25 edits" are exclusively my own performance. We know highlighting personal performance can trigger people (think of fitness or sleep-cycle apps), but we also know most lose interest very fast.
2. How to measure feelings? Lets say motivational feedback makes me think more positive of my editing experience. How to measure this? The number of edits I feel need to be done is not going to increase just because of a computer-generated message I received. How could it? I know what I know. I do have my fields of experience I can contribute to. I see what an article lacks. I only have so much time on hand. None of these factors is going to change because of a pop-up on my screen. It just makes me feel better.

A general comment: I don't see how this question directly relates to the 2 current focus areas.

As a newbie, I agree with TMg. I don't find the automatic messages "Congrats, you've must made your nth edit" especially motivational but they're good anyway. They don't influence my level of activity. But they do give me feedback on my (quantitative) level of 'engagement'.

I've found personal feedback from other wikimedians far more motivational. This ties in with Opportunity 15 in the REBOOT report. Mikemorrell49 11:45, 12 December 2017 (UTC)

I have a negative view of sites that send 'milestone' messages like these. I find it annoying and condescending when software chatbots to interrupt me to pat me on the head for reaching N actions.

There's also an extremely annoying bug. The milestone counts are tracked separately for each wiki. If some task takes me to one or more other wikis, I get spammed with several worthless 'first edit' notifications. I have thousands of edits. I seriously don't need to be congratulated for my first (or Nth) edit on each of the hundreds of projects/languages.

Positive feedback can take many forms and serve different purposes.

In terms of form, the example of the study focuses on surfacing personal productivity, but surfacing the positive impact that your results have for others can be more effective. This short clip from an interview illustrates well the joy of randomly finding that an article you created grew over time and was seen by thousands of people.

In terms of purpose, increasing editing activity may not be the only valuable goal. Making users more motivated so that they don't leave when they receive the less positive feedback, or helping to understand better how the community works are also legitimate goals.

For example, in a new notification type I proposed, the messaging touches some of the points I mentioned above: "Congratulations. Your contribution on X has been live for a week, with 154 views and new contributions from 7 others.".

I really like the notification type that you have proposed. I think that shows impact, and not just amount. Have we tested that? On Twitter, I'm always happy to see how many times my tweet has been retweeted. I imagine this could be similar.

Thanks for your comment, @MKramer (WMF). No, this has not been tested. It would be great to test a small experimental initiative in this direction and measure if it has any impact to encourage short term participation and/or longer term retention.

It might be worth encouraging personal feedback, at least to the extent of producing more "Bob thanked you for your edit" messages. I sometimes make an edit at a wiki whose language I can't read. "You made your first edit" is worthless in that situation. "Thanks" tells me that the edit was checked by someone else and wasn't impossibly bad.

Finding what to do as a new user is one of the aspects that intersect with different issues surfaced by this research. Several ideas have emerged around the area of task recommendation/suggestions. So I thought it would be relevant to share our experience providing suggestions of articles to translate in Content Translation.

We wanted to suggest articles for editors to translate, and we wanted those suggestions to be relevant. The Language team worked with the Research team to integrate their recommendation system, but before the integration was ready the Language team used a very simplistic approach as a placeholder (showing featured articles that were missing in your language).

Graph showing the initial increase since suggestions were introduced to Content Translation, and the spike resulting from improving the algorithm.

At some point we changed the basic suggestions algorithm to the more advanced one from Research. Content Translation was sending the recent translations from the user and getting relevant suggestions related to those. During user research around that time it was noticeable that the more advanced recommendations were working much better according to the users perception. But it also resulted in a big spike in the number of suggestions that users picked to create an article, going from under 50 per week to more than one thousand (the graph of that initial growth is shown at the side).

Since then, the number of suggestions selected by users to start a new translation has grow to more than ten thousands per week. We still don't know which percentage of those end up becoming successfully published articles, but our current data indicates that we at least are able to provide suggestions that are relevant enough for users to be interested in contributing on topics they may not contribute by themselves otherwise.

I just wanted to share the story to highlight that we have a powerful recommendation system that has proven useful in a specific context such as translation. Knowing that we are able to provide good recommendations may help to reduce the uncertainty when evaluating the risks of projects in this area.

Thanks Pau. I have always felt that we don't do enough to enable users to pursue their natural subject interests. Clearly, this recommendation system works because it does that, suggesting tasks that are aligned with areas where users have demonstrated interest.

There was some work (five or more years ago?) on suggesting "easy" editing tasks. One outcome was that new editors would much rather edit the page that they're reading right now, than to go edit a different, easier page. We got more editors, but since they were likely to be reading popular and/or controversial topics, then they got more conflict.

I think the work @Whatamidoing (WMF) refers to is the Onboarding work the Growth team did. In particular, the last experiment, OB6, where users were given the choice to either improve the current page or get random easy tasks. On the metrics meeting of November 2013, there was a short presentation about this (jump to minute 21 of the video) with more details.

My interpretation of the results is that many users create an account having a specific editing task already in mind. They may want to fix a typo on the current page or create a new article about their favorite music band. In any case, proposing them to do something else is not effective at that point.

In addition to quality, the way and the moment in which suggestions are presented have an impact. Compared to the approach to suggestions used in Content Translations, the onboarding experiment was not providing a persistent point for users to return to and get suggestions later when wondering what to do (since the focus was on the first-time experience) or connecting those suggestions with the topics of interest for the user in any way. Both of those aspects were considered in the case of Content Translation.

An interesting update regarding the impact of suggestions in Content Translation. During the last quarter (known as Q2 or just the Oct-December period) we saw that about 20% of new translations were started from the suggestions offered. Among other things, it would indicate that suggestions exposed a significant amount of interesting topics for users to translate they may not have thought of otherwise.

Another way to put it: providing personalised suggestions resulted in a 25% increase over the translations that users proposed by themselves, a significant number of translations that may not have been started otherwise.

What, specifically, are the goals of the New Editors project? And what are the numerical measurements that will help us track success on those goals? Daisy, Pau and I met today to try to hash out that latter question in particular—partly because putting metrics tests in place takes time.

Nailing down the  specific goals of the New Editors project is not  as obvious as it might seem at first. After a lot of discussion, we’d like to posit that that the goal of the New Editors project should be to help new editors progress from a state of wiki ignorance to some (definable) state of wiki experience.

In other words, it’s not our brief to make people experts editors, or to ensure that they contribute over the long term—as valuable as those goals may be. The interventions we pick should be aimed at getting users through their initial period of vulnerability, over the hump, out of the danger zone—to successfully deliver them, in effect,  OUT of the state of being “new editors” and into some level of mastery (where they don’t know all the answers but they know how to figure things out). If we agree on that general goal, the questions become: what level defines the upper limit of “newness” for an editor, and how do we measure progress toward that limit?

Here are some ideas for measurements that will help us track users’ progress from ignorance to basic mastery (based on Google’s HEART framework). We're not suggesting any level of improvement on these as a “goal” yet; they are simply indicators of success worth tracking.

Engagement

• % of registrants who make their first edit (within a defined timeframe?)
• # of editors monthly who make it past their first 10 (non-reverted?) edits
• # of new editors monthly who make it past their first  50 (non-reverted?) edits
• # of new editors monthly who make it past their first  100 (non-reverted?) edits
• # of accounts created (not our focus at this point, but still relevant)
• # of edits a new user makes in the talk (or user talk?) namespaces (not a goal itself, but a good diagnostic tool, since communication has been shown to correlate with success)

Retention

• % of editors who are active in the second 30 days after their first edit
• And/or, the overall # of editors who are active in the second 30 days after their first edit

Task success

• % of edits in the users’ first 30 days that are not reverted.
• % of edits in users’ second 30 days that are not reverted.

These are just first proposals for general new user metrics. I’m sure there will be lots of discussion, and other metrics will need to be established that measure particular interventions we embark on.

And we didn’t talk about the other two elements of Google’s HEART system—”Happiness” and “Adoption.” The latter corresponds with recruitment, which we’ve agreed shouldn’t be our focus until a later point in this process.

Happiness is something that we probably should create performance indicators for, but these will likely be assessed by other means than site metrics. E.g., it would be quite useful to track users' answers over time to questions like: “When you try a new task on wiki, do you feel you know how to find out what the policies around that activity are?”  But we can save that for another post. Thoughts?

It sounds like you're trying to decide where the dividing line is between "new editors" and "not-new editors". Do you think that any of the existing definitions, such as m:Research:Metrics or m:Analytics/Metric definitions would be suitable for that?

Note that ACTRIAL is in progress on EnWiki until March 14th 2018. It may have a significant effect on relevant metrics. It may be a good idea to avoid any data period which spans that date.

It's also notable that during ACTRIAL, new users who wish to create a new article are funneled to make those initial edits in draft space. Metrics for "productive edits" usually exclude edits which are reverted, or page-deleted, within a few days. Edits in draft space are unlikely to be reverted, and unproductive new pages will typically be deleted after half a year rather than deleted in a few days. It would require an unreasonably long time frame an automated system to try to distinguish productive draft edits from unproductive draft edits.

@Alsee, thanks for pointing that ACTRIAL can produce some turbulence in the data for English Wikipedia. In this case the main focus of the New Editor Experiences research is on mid-size wikis, so I'd not expect measurements and initiatives to focus on English Wikipedia. Many findings and solutions probably apply to English too, but it should not be the main or only focus.

In any case, I think the sooner we define and start monitoring the key metrics for the project, the easier it will be to establish solid baselines that account for different variability factors (seasonality, editing peaks due to campaigns/news, other community experiments, etc.).

OK so, I do not think there's a good distinction between retention metrics and how well newcomers move from "newness" to non-"newness". Retention can be a very fine-detail metric. E.g. what proportion of people came back and did *anything at all* in a second session on Wikipedia while logged in?

I think Alsee made a great point and that we'll need to be wary of the basic "productive edit" measure. I think that if we're not looking for a certain proportion, it'll be a useful metric to track -- especially in an experimental setting where everyone is equally directed to draft-space.

Generally I'd like to step back from defining metrics before we define what "success" is. It seems that Joe is trying to push back against the notion that success is more good edits or more retention. I'd like to explore this more. What's lacking in these operationalizations? E.g. if newcomers because non-newcomers but then leave anyway, is that success? Can you imagine an instance where a newcomer saved more good edits and has longer retention that is undesirable?

Aaron asks: "if newcomers become non-newcomers but then leave anyway, is that success?" I think the answer is yes, it could be. But let me turn that question around: if a higher percentage of registrants than in the past make it (just to pick something hypothetically) past their 50th edit within a year, but we see no net increase in the ranks of active editorship, is that a failure for New Editors? I don't think so.

My point is this: at various points along any editor's journey, a variety of factors might cause the editor to drop out; interventions could be devised to reduce the risk at each of these junctures. But the New Editors research looked only at the issues that affect editors in their early days, and the interventions we've discussed are designed specifically to get users past these initial challenges. We don't know what interventions would help sustain editors over the long term, and we're not considering such projects.

In a mars landing, if the rocket escapes earth but the lander fails somehow, everyone is sad but no one blames the launch team, who met their objective. That's what I'm "pushing back" on: we should measure lots of relevant indicators, but I want us to pick goals that are appropriate to the the scope of of our activities.

Actually, here's a better analogy than my mars landing example. Pre-school programs improve grade-school success. It's likely that some higher proportion of those successful grade schoolers (who went to preschool) also go on to college. Which is great. But no one, in setting up a preschool program, would set the goal as more college matriculation. Not because they don't want it, but because too many things can happen between preschool and college that are outside the control of preschool administrators and teachers.

I understand what you're saying, and I believe Aaron does as well. I believe what @Halfak (WMF)'s question was getting at is that we already have good evidence of what kind of newcomer behaviors (at what point in their lifecycle, and for what duration) predict "wiki mastery", because they are the same ones that strongly predict their likelihood of retention over the long term. You don't become a wiki master without sticking around, and vis versa.

And we know you can predict long-term retention really well from relatively short term activity measures. For example, if someone is still editing a week after they've started, they're very likely to be editing a year later. A month in, they're even more likely. We've been calling these "retention" metrics. But you can call them whatever you want, but let's use existing metrics and thresholds as our baseline and starting point. You can and should add to these, but don't ignore or replace them.

Thanks for clarifying Jonathan. I'll try to clarify what I'm interested in as well, since I think I've touched on a number of ideas.

My main goal here, as I think about it, is tactical. I want to come up with a list of indicators that would help us measure new editor success. Then I want to write some Phabricator tickets to ensure that we are able to get those measurements easily—perhaps even on an automated "report card" like those that exist for various products and constituencies.

My thought is that we can do this AHEAD of starting work on any particular interventions, so that when we propose and then create those interventions, we'll know what type of influence we're trying to have. The way things typically go, by contrast, is that I'll propose that a project should "grow X," but I have no idea how big X is now, so I don't have any particular target. Getting to any destination is a lot harder, needless to say, when you have to make a map of the territory first.

You suggest "let's use existing metrics and thresholds as our baseline and starting point." Sounds good to me. So, what should we be looking at? What's your list of metrics that we should be gathering. Which metrics don't make sense in my list above?

Korean barely uses the draftspace (complete list of draftspace amounts to 0.07% of articles), and none of the last 500 pages created by new editors used it.

The Czech wiki doesn't have a Draft: namespace.

It might be useful to say more bluntly what Pau appears to have tried to communicate earlier: This is not about the English Wikipedia. ACTRIAL, for example, can be ignored, because that is only happening at the English Wikipedia.

English Wikipedia is not the only Wikipedia that restricts new article creation by user class. Not sure if other wikis have Draft. Regardless, if we're building solutions to improve new editor experiences, we should be wary of creating solutions that won't work on some of our biggest wikis.

I think that it's helpful to use this HEART framework to brainstorm and prioritize metrics. I hadn't been aware of it; thanks for sharing!

I think you might want to look harder at "adoption" metrics in order to get at new users' level of wiki experience. I'm not talking about adoption of new features that you are testing, but rather adoption of features of the wiki that experienced editors use extensively but new editors don't use very much. One example is article talk pages (which you currently have as an "engagement" metric). Others might be creating a user page, using edit summaries and policy citations, or making edits to the Wikipedia namespace. "Adoption" here means that the new user has learned how to (inter)act on the wiki in a similar way to how veteran editors do it.

I see where you are going with your engagement metrics, but given the timescales you're using (months), these look more like retention metrics. I believe that "engagement" metrics make more sense on smaller timescales: for example, edits-per-editing-session, or length-of-edit-session.

Ultimately, I think that retention metrics make more sense if we're trying to show impact, especially if we're trying to measure the impact of specific interventions--for example, a new onboarding tutorial that is offered to a sample of newly-registered editors. Length of retention is the best proxy we currently have for "wiki mastery".

Two suggestions:

1. I think the posited goal should be broader than "helping new editors progress from a state of wiki ignorance to some (definable) state of wiki experience". The current goal also implies that the Wikiproject working environment (information, learning, editing, communication, collaboration, tools) is pretty much OK. Incremental improvements are ongoing. The posited goal focuses on helping new editors (in better ways) to learn about - and to work in- this environment. IMHO it's just as likely that the characteristics of the working environment are a factor in editor retention. Various newspaper articles on 'the decline of wikipedia' mention this factor. In both the REBOOT report and the 2017-2018 Plan (Program 4) the goal is stated as: "Improve the new editor experience". This is a broader definition.
2. To track progress towards this broader goal qualitative measurements of new editor experience (like survey data) are needed. For the quantitative measurements, too much focus on 'number of edits per time period' gives IMHO a too narrow view of what's really going on. I suggest including things like:

• number of new pages proposed, accepted and rejected
• number of existing article/media items edited (with types of edit, major/minor, etc)
• number of article discussion pages edited
• number of user talk page pages edited
• time spent logged in per month

Hope this helps! Mikemorrell49 (talk) 14:44, 12 December 2017 (UTC)

One more thing ...

Google's HEART framework looks very applicable in this context. Very useful too! But I would avoid 'cutting corners' in using the framework. The framework is designed so that goals can be defined for each of the HEART components. Then 'signals' (indicators?) can be defined for each component that indicate the extent to which specific goals are being achieved. And lastly, these indicators are translated into more detailed 'metrics' that can be measured and together provide an update on the 'signal' (indicator).

At the moment, we have just one overarching goal and various metrics for 3 of the 5 HEART components.

I strongly suspect that the HEART components are sequentially interrelated. If I'm right, measuring just some of them independently will not tell us what we need to know. For example, A high level of 'Happiness' and 'Satisfaction' is a necessary condition for Engagemement. A high level of Engagement is a necessary condition for Adoption, and so on. Again, if I'm right, the whole process of retention and task success starts off by ensuring that new editors are 'Happy' and 'Satisfied': with the Community, with their interaction with the Community, with their 'onboarding' and with their working environment. New Editors who become - on balance - 'unhappy' or dissatisfied with these are unlikely to become Engaged. The first finding in the REBOOT report was that new contributors sign up with different motivations and expectations. 'Happiness' and 'satisfaction' is not uniform but related to these individual motivations and expectations.

Metrics that together indicate the personal level of Happiness/Satisfaction, Engagement, Adoption, etc. are IMHO much more important than measuring 'number of edits'.

Many years ago I worked in Human Resource Management and Development. Topics like 'Motivation', 'Learning', 'Collaboration', and 'Having the right resources' were as hot then as they are now even if the technology has changed a lot since then. There are multiple theories and models about 'motivation' that all have a degree of merit. Personally, I found 'Herzberg's two-factor theory' to be one of the most useful in practice. He distinguished between factors that - if present - led to positive motivation and other factors that - if absent - led to dissatisfaction. Maybe this distinction is useful in surveying new contributors.

Thanks for pointing me to Herzberg's two-factor theory @Mikemorrell49. It's a useful lens for looking at the various ideas we've entertained to improve conditions for new editors. I see now that most or our proposals are aimed at reducing dissatisfaction (by providing better information, more efficient tools, etc.), while only a few are designed to produce more satisfaction (by providing recognition, for example). Luckily, the work itself of creating knowledge is meaningful to many, and thus has high intrinsic value.

Speaking of intrinsic satisfaction: High edit counts are trivial to measure, but they don't necessarily correlate with high satisfaction. Reverting spam or slapping "citation needed" templates on articles is not the most satisfying activity (for most people). But creating significant, satisfying content can often be done in a couple of edits.

Perhaps the list of "success" metrics should include measring the volume of content added (ideally excluding reverts, as reverting page blanking gives spurious ideas of how much content you "added").

Answer is: not really. Ranking the personas numerically would require a different type of research.

The personas you devise in the “Motivations for Editing Wikipedia” section should, indeed, be  quite useful for understanding what drives users. There’s one thing I wonder if you can add that would make these even more useful, especially for product people like me.

I’d like to know which  of the six personas you think are the most common? This might help us understand whose problems to aim our products at, especially if we want to have the biggest impact. Such info can also be useful when we’re advocating for or explaining product choices. (For that reason, it would be most desirable if you could add remarks on the subject to the actual report, if possible—for citation.)

You don’t need to break out the statistics; just say which personas, in your estimation, are the the top two or three, perhaps? A ranking would be most welcome, along with any insights you care to pass on, of course.

Thank you for your note, @JMatazzoni (WMF) :)

@Neil P. Quinn-WMF and I could put our heads together and go through the list of all the participants, and add up which personas are associated with the most respondents. However, I am not sure this would be an accurate representation of the numbers, or priorities you are asking for. What you are asking for, needs proper statistically significant research. Qualitative work like this contextual inquiry, is not statistically significant.

However, there is work that could be done, to help us better understand the numbers of people coming to various wikis, their motivations, and perhaps, how many fit under each "motivation bucket". Think of the "Why we read Wikipedia" research that @LZia (WMF) and team did about the motivations of people reading Wikipedia. @LZia (WMF) has proposed similar research for "Why we edit Wikipedia", and if that work could get done, we would be able to answer your question with more confidence.

All this said, I think we can answer the question about which persona is most important to focus on, if we think about the problems (or opportunities) we want to and are able to address (from the findings of the research), and then ask ourselves, which personas we need to focus on for that solution or opportunity.

Another thing to remember, is that while we focus on new editor experiences and needs, we also need to very closely, keep in mind the experiences and needs of experienced editors, curators, page patrollers, and all the contributors on wiki who are already working with new editors, and otherwise deal with the work that comes with more new editors. In order to keep these people in mind as we work, we have pragmatic personas, and to fill in the gaps (since those personas are "pragmatic") we will most likely reach out in various ways to ensure that our goals of retaining more new editors, will not impede or poorly impact the work of experienced contributors.

Any chance the Wayne persona can be updated? I don't think the VE concern (cited on the on the last line) was ever that common, and it must be pretty much nonexistent by now.

I can think of a lot of replacement concerns. A good catch-all would be 'Wishes there was more community input before new products are built'. If you want something VE-specific 'Rarely finds VE useful, believes new users are better off with wikitext'.

P.S. The persona cites Kindle and Android. However whenever mobile issues arise a chorus of complaints arise from this persona. They've never even seen the mobile-only feature before, it's effectively invisible to us. People freak out when the WMF adds invisible mobile-only content we can't see or fix. (Such as the Related Article feature, and there's currently a storm over Wikidata article descriptions, and I bet almost no one knows you're pointlessly posting "Last edited by username" at the bottom of mobile.) It seems that the typical "power editor" generally considers desktop the only suitable platform for real work.

Hi @Alsee,

Thank you for your note. I agree that Wayne and all of the pragmatic personas should be updated, and will remember your suggestions when it happens. It is helpful to hear your feedback on mobile, specifically helpful for iterating the persona, and useful feedback in general for WMF.

Just a point related to the third characteristic of New Editors described in the report which noted "People​ ​do​ ​more​ ​complex,​ ​rigorous​ ​tasks​ ​on​ ​their​ ​desktop​ ​or​ ​laptop​ ​computers,​ ​and​ ​short,​ ​quick tasks​ ​on​ ​their​ ​mobile​ ​phones." (p9).

On the one hand, I wholehearted agree that we should introduce more 'micro-contributions' opportunities on mobile for users (e.g., see this old discussion of Android contribution concepts Reading/Readers contributions via Android).

However, given that "10 [out of 47 study participants] contributed to Wikipedia using their mobile phones or a tablet...(one used the iOS app, and one used the Android app); the majority of these editors were young (under 35 years old)", couldn't this be seen as a trend towards people wanting to edit more on mobile in general and not just small tasks? I found that 10 out of 47 to be arguably quite high (especially the 2 people using apps to edit when they are a relatively low % of overall Wikipedia users and apps have very minimal editing functionality), and it can only increase given the user base will shift more and more towards having mobile as the primary device.

I wonder if it may be a chicken-and-egg situation that users are not performing more complex tasks because mobile editing capabilities are currently quite limited (eg., Visual Editor is presented with reduced functionality on mobile web and not available at all in the apps; it is not possible to create a new page in apps and mobile web on phones).

TL:DR; Are there opportunities to encourage engagement of New Editors through overall improvement of mobile editing capabilities which could be explored in further research and design?

You make some interesting points. And I think there is a lot of potential for creating more granular tasks that could be handled on smaller screens.

On the other hand, it's also true that the tasks that don't get done in a backlog are frequently those that are just harder to do, because they require a lot of rewriting and research and expertise. So do we help people do what's already easy and fun, or do we help make what's harder easier?

The answer is that we have to do both, of course. And it's possible that part of the answer will be to find better ways to break up bigger tasks into smaller tasks. E.g., when I edit an article IRL, for example, I don't just hand it back to the author and say "this needs cleanup" or "this has grammar problems." I underline particular sentences and words and say what's wrong with them. I think we need to get to a system, somehow, that is more like that.

In the Netherlands, voluntary 'micro-contributions' are the trend in voluntary work generally. In the past (10-20 years ago), volunteers 'joined' a voluntary organization and stayed loyal to it. They were recognized for the number of years they had been a volunteer. That's still true for the 'older generation' but it's dying out. 'Younger generations' (say under 45) are much more likely to look for short-term win/win matches between their current interests, available time and voluntary opportunities. Many voluntary organizations have had to adapt to this trend. For one thing, the time that short-term, opportunistic volunteers are prepared to invest in learning is less than long-term volunteers. So short-term volunteers tend to work on small chunks of work under the supervision of more experienced and better trained longer-term volunteers.

Although the specific projects have not been defined yet, I think their ultimate goals are clear: improve the acquisition and retention of new editors. Regardless of the type of project and approach, it will be very useful to have metrics and instrumentation to measure the impact on such goals.

This will allow to establish a baseline first, and then evaluate the impact of any initiative in this area. I created a ticket to capture the proposal in case anyone is interested.

Hi @Pginer-WMF, I agree. Personally, I see metrics and instrumentation as essential. Thanks for being pro-active in suggesting this! To be honest (because I'm a completely newby in most things) I'm not sure what happens next. Would you very briefly explain what a (wikimedia) ticket is? In general/ICT terms I know what it means. Just not what it means in the Mediawiki context. A link to somewhere where this is explained would be great!

I read a lot about the consensus approach to moving forward with anything. Would you (for me as a newby) just briefly summarize how tickets gather support (or not)?

Many thanks for your initiative and for your patience with a newby!

Mikemorrell49 (talk) 16:33, 23 November 2017 (UTC)

An item on Phabricator is a task or ticket. Phabricator is a site mainly for things that directly or indirectly require programming or configuration changes. It's mostly used by Foundation staff and volunteer techies, but it's open for anyone to submit new items or to discuss how tech issues will be resolved. Few editors/administrators venture onto the Phabricator techie-zone. (Note: Admin are editors, promoted to admin by other editors. A very small number of admin also happen to be staff.)

Your question about consensus is complicated, chuckle.

The specific ticket here, T181249, is a simple case of Foundation staff wanting metrics and instrumentation. Tech staff will handle that just like any conventional employer-employee authority model.

There are also a huge number of small uncontroversial fixes and improvements made by the Foundation every month, without dealing with consensus.

If anyone posts a valid bug report, it will generally just get fixed. No need to discuss consensus.

If the community wants something new or changed, the primary discussion is usually elsewhere. However the task generally needs to end up at Phabricator for tech-action. Depending on what the task is, a common case is that Foundation wants evidence of community consensus for that task. There are a lot of tasks that have been "open" for years, but simply haven't had any programmer time devoted to them. I also have an interesting link to some community-consensus-requests that were rejected by the Foundation.

The thorny issue is when the Foundation wants to deploy or change something major, and the community considers it harmful. Everyone has good intentions, we all want to work together, but sometimes the two sides have different perspectives on the best course of action. We have been unable to agree on a process or answer here. I could list some conflicts that have ended in various ways, but this post would get unreasonably long. There has been longstanding and very serious tension between the Foundation and the community on the question of deployment-against-consensus. The question itself leaves us in a state of constant and toxic tension, even in the absence of any active dispute. Both sides believe they have moral authority, and as a practical matter both sides have the power to catastrophically nuke each other if a conflict ever escalates out of control.

As on many other unrelated points, thanks for your response and your helpful info, Alsee. I've seen one or two examples of differences of opinion and of processes to ask for and give - or withhold - support for proposals. I take your point about tensions between the Community and the Foundation and I can imagine situations that led to these and maintain them. Something for me to be aware of. At the comment, I have far too little knowledge or experience to comment. But it does sadden me to hear that mutual cooperation and trust is far from ideal in some cases.Mikemorrell49 (talk) 13:04, 26 November 2017 (UTC)

First of all, my compliments on this initiative, the quality of the research and the relevance of the opportunities mentioned! I wholeheartedly support the conclusions. If I can contribute in some small way, please let me know. The reported experience of new editors matches mine (a newby of 2 months). I would be delighted if some headway could be made in leveraging these opportunities. I also see some links to the Strategic Direction 2017. I'd like to add 3 suggestions:

1. In the report, it would IMHO be worthwhile adding 'education level (or equivalent)' to the demographics of the survey group. The reason is that - based on a couple of my random samples - the 'readability' (and therefore the 'editability') of at least some Wikipedia articles are at a level consistent with a college education (US grade 13-15). Perhaps this is what we want, perhaps not.
2. Good to read in the conclusion that there are other types of 'editor' roles in which people can contribute and collaborate other than in the general 'editor' role. As part of this, I suggest a possible opportunity in improving the way a) interests and skill sets (at the discretion of contributors) can be standardized (as are language skills now) and b) the way in which contributors can request collaborations based on interests and skills.
3. Findings 5 (complexity) and 8 (conceptual challenges) focus on editing content “the Wikipedia way.” I fully agree with these findings and opportunities. From my personal experience the complexity of - and conceptual challenges to - understanding and contributing to policies, strategies, processes/guidelines, the organization of projects (large, small, inter-) etc. is just as challenging and perhaps even more so. This point is addressed throughout the report. Perhaps it is worth capturing a 'synthesized opportunity' in improving the way in which newcomers can participate in non-content related discussions.

Kind Regards,

Mike ~~~~

@Neil P. Quinn-WMF Is there a preferred way for referencing this research? I make up something if I will devise something if I don't hear from you in the next 12 hours. :)