Talk:Growth/Personalized first day/Structured tasks/Add a link

Tracked in Phabricator
Task T326791

I typically dislike when tutorials auto-advance from step to step, as sometimes I may just be reading something more cautiously, or I may be momentarily distracted with some other task.

For context, I'm an experienced user, so not the target audience for this feature, but adding links between articles is a type of edit I always liked to do, so I enabled the newcomer homepage so that I could try it out, and was quite surprised when the tutorial auto-advanced without warning.

I would recommend reconsidering the decision of enabling the auto-play by default, and instead using some other nudge like flashing the next number in the tutorial (after a reasonable delay), so the user will know to click it when they finish reading the instructions.

Thank you for your feedback. We went with auto-advance as a way to show that next steps are available.

I'll check on the data we have regarding how steps are read now and if people play with the backward/forward buttons.

@Waldyrious, we will observe the interactions users have with the auto-advance and from there decide on the possible actions. This task is documented.

so we found in hewiki, that wrong links suggested by the tool are often added by people using it.

we would like to "strengthen" it a bit by adding or modifying the instructions the tool presents to the users.

can you please point to the "system messages" used by the tool? or better yet, could you explain how i can activate "add a link" tool on one's own account, presumably by adding something to the personal JS ?

currently, some but not all users see it on hewiki.

peace.

Hello @קיפודנחש.

To enable the "add a link" tool, you will first need the newcomer homepage enabled, and then you might need to complete a second step if you haven't been opted into the "add a link" experiment by default.

1. From your preferences, you will want to be sure that הצגת דף הבית למתחילים is enabled / checked.
2. Then navigate to your newcomer homepage.
3. Select some topics.
4. Then if you only want to view "add a link" edits, then you can select עריכות קלות and make sure that only the following is selected: הוספת קישורים בין ערכים

Are you not seeing that option? I can provide a snippet of code if needed, but I think once you enable the newcomer homepage you should see that option.

To help new users utilize "add a link" more effectively, I also highly suggest you customize the Growth Configuration for hewiki. There are many improvements that could be added including adding a "Destination page for learning more about link recommendation task type."

Does this help?

@KStoller-WMF: yes, it helps. thank you so much!

peace.

Hello!

Among many overlinking incidents this brings, there is one that is very common across many pages on ruwiki: in the bibliography section the tool suggest wikifying the city where the publisher resides (most commonly in ruwiki that is Moscow (Москва), lots of links, at least in the (rather small) selection of articles I encounter). Unfortunately, it is not obvious for newcomers that those links do not really add value (moreover, I myself am also more or less a newcomer and I am not sure, whether there is actually a consensus established against this behaviour, but at least it seems like there is none for it). It seems logical to stop this from happening.

One possible way would be to have an option of disabling this function for certain names of sections, configurable per-project-wise. This way we could, for example, disable it in bibliography section, this would have a negative implication (adding wikilinks to authors might be net positive), but it would stop this flood of links to cities.

Another possible way could be having a blacklist of articles, to which links should not send.

Sorry, now I see that this is already discussed in T279519

Hello

Do you have some diffs I can use for documentation purposes? It would help to prioritize the work on Add a link, by having a better understanding of the context in which links have been applied.

Feel free to post them directly below this message, or to comment on the Phabricator task.

Thank you!

The UI of adding the links is the really great part of this feature! It is simple and quick.

Unfortunately, often the process results in several links in one diff, out of which some are overlinking. When a patrolling member visits the diff, they might want to revert part of them. Obviously, it is not very hard to revert several of them by hand for any particular page, but there are many of them so it would help if this process was somehow streamlined.

I would suggest having a version of this tool accessible to the patrollers, where they could review the changes and, in the same interface as the newcomers, cancel some of the changes. The additional profit is that the tool would get the feedback on the reasons, why this particular link should not be wikified.

Thank you for your message Medvednikita.

Coincidentally, we have discussed about this issue in a team meeting more or less while you wrote this message. Having a tool inspired by add a link for patrollers (or, why not, for more experienced newbies) is something we could consider, but only if it is a popular request.

This is a small and unimportant problem, but it still felt strange, so let me describe just in case. When you finish going through suggested links in an article, there is a button to submit, and a button to return to the editing process -- but no button to cancel all the changes and go pick another article. This is not a real problem -- one can obviously return to the previous page or use any other way of navigation -- still seems nice to simply have a discard changes option.

Hello Medvednikita

Thank you for your feedback!

What would be the use case there? You'd like to have a "cancel" button because you aren't sure of what you've done? Because you were just testing editing? Another reason?

I'm curious of the motivations there, since it would help us to understand the entire process a user follows in this case. ;)

The role of Suggested Edits is to teach users how to edit. But this task is an unusual way to edit, that do not teach the users how to add a link in a normal way.

I think that this feature will confuse newcomers. The newcomers should learn how to add links, not to decide if the suggested links are ok. When they will normally edit an article, there will be no AI to help them.

Hello @NGC 54 -- thank you for thinking about this and posting your opinion. When we first started talking about structured tasks, we thought about this exact question: should we aim toward teaching newcomers to use the traditional tools, or toward newcomers being able to do easy edits at higher volume?

I agree that the feature we're building is a new way to edit. We think that this is a good idea for two big reasons:

1. We know from our research that many newcomers struggle to succeed with editing at first, and we think that if they have a quick way to be successful, they will feel more positively about editing and want to continue to do it.
2. We also know that many people are unlikely ever to edit using the usual tools. Perhaps they want to contribute, but don't have time to learn or think that using the usual tools is difficult on their mobile device. We want to try building a new way to edit, which will hopefully allow new kinds of people to edit.

For those reasons, we decided to build a new kind of editing flow, instead of only showing newcomers how to use the existing tools. But that said, many community members brought up a similar point that you did: even if we make it easy for newcomers at first, we want to make sure that they can learn how to use the usual tools when they are ready. We don't want them to get stuck with link edits, but rather to move on to learning more.. That's what our team will be working on next, in a project called "positive reinforcement", in which we will try to encourage newcomers to continue editing and try more challenging kinds of edits.

Does this make sense? What do you think?

@MMiller (WMF): I do not think that adding a link is a very hard task. Remember that VisualEditor shows you how to add a link when you open it for the first time (if I remember correctly)...

In any case, the normal task that is currently deployed will disappear?

@NGC 54 -- I think that your opinion is valid here; it's possible that this structured task will not make a major difference to how easy it is for newcomers to edit. But we think it has that potential, and we're going to be collecting the right data to be able to tell whether our hypothesis is correct.

Yes, this will replace the previous unstructured "add links" task when it is deployed. But we do plan to keep a control group of users who will continue to receive the previous task so that we can compare and see whether outcomes are different for the users with the structured task.

I have seen this tool only be used by new editors, editing 60 pages a minute. That means they are only adding links without knowing what they add, and without adding value, actually adding nonsense. The links they add are technically valid, but are usually distracting the reader to non value adding information without any sensible context. To my blunt opinion this tool only leads to vandalism.

Hello @Edoderoo -- thanks for coming here to weigh in. Your experience is from Dutch Wikipedia, correct? We're glad your wiki is one of the first to try "add a link", and I appreciate that you looked at the edits and provided this feedback. The feature is only deployed to about 10 wikis right now, because it is in the phase that we call "Iteration 1". This is the first version, which we use to learn and improve for future versions. When we deploy a first version, we expect to see problems, and we also hope to see value -- enough to make us want to keep working on the feature for future versions.

Our goal with "add a link" is to give newcomers a tool that makes it very easy for them to make their first edits, so that they get excited about Wikipedia and want to learn more and make more kinds of edits. We know that wikilinks are not the most valuable improvement to an article, but we thought that the value is that it would be something simple to get newcomers involved, and something with which they could not do too much damage. We now have enough data to see that the feature actually accomplished this goal: newcomers who have it available are more likely to make their first unreverted edits to articles than those who don't. In other words, it causes people to edit who never would have otherwise. That is good news for this idea.

But as you say, there are also issues. Some newcomers use the feature so heavily that they are overlinking. Other ones may not be applying strong judgment. We've heard similar feedback from Arabic Wikipedia, Hungarian Wikipedia, and German Wikipedia. These are some of the ideas we've gathered so far for improvement:

• Nudge newcomers to do other, more valuable kinds of tasks after they do a few link tasks. One task we might nudge them toward is "add an image", which we just released on Arabic, Czech, and Bengali Wikipedias (I would be very interested in your thoughts on that task).
• Don't let newcomers do too many of the tasks or proceed through them too quickly. We might only let them do 25 per day, or stop them if they are spending less than, say, 30 seconds on an article, or stop them if they are saying yes to more than 90% of the suggestions, etc.
• Don't offer so many links per article -- right now we offer up to 10 suggestions per article, but we might limit it to 3.
• Don't allow links in sections that usually shouldn't get them, like the References section.
• Limit the suggestions to articles that seem underlinked, perhaps by looking at the ratio of wikilinks in the article to bytes in the article.

What do you think of these ideas? Can you think of other ways that we can keep the good parts of the feature (causing more newcomers to edit), while reducing the bad parts (generating some low value edits).

We'll be working on the improvements in the beginning of 2022. I look forward to hearing back! And if you are able to get opinions from any other Dutch volunteers, those would be valuable.

Right now I see links being suggested like Nederland which is way too much of an open door. A link should add value, like an additional edit should add value, not add bytes. Adding links is quite a delicate task, and I doubt it is a good thing to hand over to newbies. Let me be blunt: a stupid script is suggesting links, and an unexperienced (read: still stupid) newbie says: 'Yes, this is a link'. If a script is needed to get newcomers in, we're on the wrong route. For me it is analog to learning kids to write through grafiti on the wall. Technically it is writing, but it is also a secure way to make sure their efforts will be seen as vandalism.

But if you want to encourage ppl do add value, then let them add one link, and challenge them for the next edit to find a source, to add a sourced fact, to find and correct a spelling mistake or a grammar thing. Images will be tricky, as most ppl do not understand our license model. What I have seen so far, is that now ppl are challenged to add 100 links to 50 pages. The next day they find several of their edits reversed, which is also not an invitation to do some more work.

Today I try to do some experimental edits in bnwiki. When I save edit's than I saw that edited article was added in my watchlist. Why this is was needed? (diff, diff, diff)

@MdsShakil -- thanks for bringing this up, and I see that you found the Phabricator task. The reason we automatically add the articles to the watchlist is because this task was primarily built for newcomers, who do not know about or understand watchlists. We wanted to add by default so that they could start building a watchlist and getting notifications, helping them get more involved in the wiki. What do you think would be best for us to do?

Also, do you have any other thoughts or ideas about the "add a link" work?

Tracked in Phabricator
Task T283985

ro:Special:Diff/14360660: „[[server]]<nowiki/>e”.

Thank you! We will investigate it. I found two cases at French Wikipedia as well.

Sometimes the tool quits without apparent reason, which means I click on the suggested next article, it opens, but the tool does not start.Usually quits after about 8-10 articles.

The window of the tool is too small. Little info is visible when it shows some text from the recommended article.

The algorithm tends to suggest tv-series or movies which have everyday life titles.

The selectable "topics" too wide, it practically allows everything. It has little relevance if I have chosen "Africa" only, the article can be a european city.

I suggest to make the suggestions more accurate.

I use this tool only for 2-3 days, and I noticed that many suggestions are incorrect, pointing to a quite different topic. Of course I did not accept these false suggestions.

However I received notices from other editors (who do not use this tool), that put links to articles "that are not relevant".

So, it would be better to have fewer, but more accurate suggestions to articles.

If it is now 75%, than please make it 90% accurate at least.

Hello @Misibacsi -- thank you for trying out "add a link". I really appreciate these detailed thoughts, especially since we haven't yet heard from Hungarian Wikipedia about how it's going on that wiki. I'm also tagging @Tgr (WMF) who worked on the feature and is familiar with huwiki.

Here are my thoughts and questions for you:

• Quitting unexpectedly: when the tool quits without apparent reason, is there any kind of box that appears explaining what happened? And where are you at that moment -- are you on the article in "read" mode? Or is the visual editor open?
• Window too small: I think you're using desktop (not mobile), is that correct? We call that window the "link inspector", and it currently displays the article title and the first sentence of the article. Are you saying you would want to see more of the article in order to decide whether to add the link? If so, how much of the article would you want to be able to see?
• Everyday life titles: do you mean that maybe if the article says, "The determinant of a square matrix is a number associated to the matrix." then the algorithm might link to "The Matrix"? I've seen that issue, and I will bring it up with the researcher who works on the algorithm.
• Topics: there are 39 topics available, and I agree that we wish they could be more specific. Topics like "arts" and "sports" are quite broad -- maybe someone is only interested in football or hockey. That's an area for improvement in the future. But it also sounds like you're saying that some articles are wrong -- like a European city under the "Africa" topic. If you see some of those, could you please post the article links here so I can investigate?
• Accuracy: we believe the algorithm is about 75% accurate. What is your experience? Do you agree that it seems to be right about 75% of the time? In general we think that is an appropriate level of accuracy so that the user feels helpful. If it is too accurate (like 95%), then the user might think, "Why do they even need my help?" It sounds like you are saying that other patrollers are seeing users overlinking, and therefore you think the algorithm should be made more accurate. Is that correct?

Please let me know!

• Quitting unexpectedly: So the tool is running. I press on the article small window to open it. The article opens, and I am in editing mode in visual editor. Only the "Add a link" tool is not functioning. It happens after about 8-10 correct opening.
• Window too small: Desktop version, yes. ''Are you saying you would want to see more of the article in order to decide whether to add the link? If so, how much of the article would you want to be able to see?'' Yes, because a little part of the first sentence is not always enough. Sometimes it is a person who has a too long personal name... At least 2x much text would be more helpful from the article. I you can make the "link inspector" window bigger, there would be more space for this.
• Everyday life titles: Yes, exactly. For example in a historical article you may read "neighboring countries". However in Hungary there was a tv-series sitcom "Neighbors", and the software suggests it for link. There is just formal similarity. It was an everyday sitcom, without relations to historical topics.
• Topics: Maybe the categories can be used instead? Or if there would be filters such as "no sports", or "no animals", or "no cars". As the car industry becomes more international, the same trademark can be manufactured almost everywhere on Earth. I do not understand what you expect. 1. I choose "Africa" topic, 2. the tool opens an article of a "european city". What should I send? At that moment only the city article is opened. Can you check what topics I chose?
• Accuracy: more or less it is about 75% relevant, and the totally wrong can be ignored. But in Hungarian Wikipedia there is a rule "Not to put too many links into an article", and "Do not put links that are not relevant". Both are a bit subjective what some editors think of them, and I received such feedback from others that "the links were not relevant", and they undo some of my edits because of this. Maybe they were right and the link was "near relevant". I do not know if I could explain it well now. So, yes, I think " the algorithm should be made more accurate". If not 95, than in-between, for example 80-85%.

Now I encountered a conflict between my selected topic ("South-America") and the suggested article ("Abe no Szeimei" - [[:en:Abe no Seimei]]). The person was Japanese (lived between February 21, 921 A.D. – October 31, 1005). He was not a world traveler or explorer. The article does not mention "America".

I have a screenshot, but it seems I cannot post it here.

Watchlist: it seems that the tool automatically adds the article to my personal Watchlist, although in my personal settings I have set "all my editing are minor edits".

''is there any kind of box that appears explaining what happened? And where are you at that moment -- are you on the article in "read" mode? Or is the visual editor open?''

I forgot to answer to these.

• No error text, no box, nothing unusual, it jumps to the selected article (by the tool), and it opens in "edit mode".
• The article is in "edit mode" with Visual Editor. I can edit, but the "Add a link" tool is not active.

I have to go back to my personal "Startpage", and start the tool again.

Sometime only the article page opens, and nothing else happens.

Sometimes the tool does not start.

@Misibacsi -- thank you for writing all these thoughts out. I know you've been talking with @Tgr (WMF) on huwiki, and he has given all the notes to me. I think you'll continue to hear back from him about any next steps. I really appreciate your thorough testing, and I think your notes will help us improve the feature in the future!

Perhaps I missed it somewhere but what is the visual difference between what the three experiment groups (Groups A, B, C) experienced? Is there a mock-up somewhere? Particularly curious of what the "unstructured" interface looked like since its revert rate was so much higher.

@Czar -- good question. Here's an explanation of the three groups:

• "Add a link" (40% of new accounts): these users all have the new structured task as their default suggested edit in their feed. In the image, you can see what a user in Hungarian Wikipedia sees after selecting an article from the feed and going through three dialogs that explain the task.

  

  Add a link experience on desktop

• "Unstructured" (40% of new accounts): the legacy task is sourced into the suggested edits feed by the presence of a maintenance template. After the user selects it, they are brought to the article where a window pops up explaining that they should add links. They can see the maintenance template at the top and also a blue dot encouraging them to click edit. But they are not guided through the edit -- they have to find their own way. Therefore, it's easy for the newcomer to do the edit wrong or try out a more ambitious change than just adding a link, leading to them being reverted. And interestingly, 25% is about the same revert rate as for the edits that newcomers make outside the Growth features in the four wikis that are included in the data. In English Wikipedia, for example, the revert rate in July 2021 for edits made by accounts less than one month old was 26%. The image shows the experience in English Wikipedia.

  

  Unstructured link experience on desktop

• Control (20% of new accounts): these users don't have any Growth team features at all. When they create their accounts, they just land right back where they were before with no homepage or suggested edits. We keep a constant control group so we can always measure how the Growth features are improving the experience.

Does this all make sense? What do you make of the difference in revert rate?

That comment that a structured task is less free-wheeling than an unstructured edit and thus is less likely to be reverted sounds like a great hypothesis to me!

I suppose it also depends on how (if?) "random" edits are patrolled on those wikis. I haven't done anti-vandalism patrol in a while but my understanding is that the tools are showing editors cases in which there is an unclear but high likelihood of vandalism, which I'd suppose is rarely the case with innocuous linking. These kind of edits can fly below the radar, leading to a lower revert rate. As a proxy for community acceptance, I think that's as good a measure as it realistically gets outside of a controlled environment.

I'm glad you brought up ORES classifications, @Czar -- it made me want to check how ORES feels about these edits, and whether it does, in fact consider them high quality. I went to Russian and French Wikipedias as a spot check, and filtered the Recent Changes feed to just "add a link" edits, and then colored them by their ORES quality classification (going best to worst with green, yellow, orange, red). I was surprised by the results: in Russian, though there is no red, there is plenty of yellow and orange, so it seems that ORES is suspicious. And in French, it's basically all yellow, meaning ORES is ambivalent about the quality of the edits.

I'm not totally sure what to make of this yet, but it is pretty interesting -- I'm tagging @MGerlach (WMF), the creator of the link algorithm in case he wants to think about this, too.

@MMiller (WMF) this is a really interesting observation (and thanks for tagging me). I know very little about the inner workings of ORES, however, I think that ORES does not just rely on the content of the edit but also uses features based on the editing-history of the user. I also believe that ORES is configured differently in the different languages. So I dont know right away how to interpret these observations. Nevertheless this is a really interesting way of thinking about how to assess the quality of the edits recommended by the algorithm. Will have to think more about this.