Topic on Talk:Wikipedia.org updated page layout/Flow

We found that the test group had a 3.64% higher probability of interacting with the page and was 1.07 times more likely to engage with the page than the control group. Furthermore, although both groups received the dynamic primary link feature, users in the test group had a 6.82% higher probability (1.09 times more likely) of visiting a Wikipedia in their most preferred language.

When phrases such as "1.07 times more likely" and "1.09 times more likely" are used, does this mean ".07 times" and ".09 times"? This sounds pretty insignificant.

I interpreted them as "7% more likely to engage" and "9% more likely" to visit. And if that is the correct interpretation, it seems like a clearer wording.

The "times more likely" reported are what is called the relative risk, probability of outcome in group 1 vs probability of outcome in group 2. The probability of the test group engaging was 1.07 times the probability of the control group. Specifically, it was 3.64% higher in the sense that if probability of a control group user engaging is 75%, then the probability of a test group user engaging is 78.64%. Hope that helps clear up any confusion!

Is there a reason to say "was 1.07 times more likely to engage" as opposed to "was 7% more likely to engage"? I think non-math folks would find the latter to be much easier to understand. Or do those not have the same meaning?

Yes, because I think that might be misunderstood as a difference because there are multiple ways to say "probability" such as "chance" and, specifically, "likelihood". So "was 7% more likely to engage" might be misunderstood as "if the probability of a user in the control group engaging is X%, then the probability of a user in the test group engaging is (X+7)%" (wrong) instead of "1.07 * X%" (right).

I would be comfortable saying that the test group was 3.64% more likely to engage and their probability of engaging was 1.07 times the probability of the control group to engage. It's an unfortunate side-effect of there being so many ways to express probabilities and uncertainty. I'll think about ways to address this. Maybe the solution would be to include a thorough explanation of these phrases in every report going forward.

I created task T140187 to try to figure out the best wording of the "1.07 times" bit.

I have problem to understand and translate " dynamic primary link feature". Maybe it's me (I read very superficially becausee I am in lunch pause) but can someone explain a little bit more? Thanks.

Although both groups received the dynamic primary link feature, users in the test group had a 6.82% higher probability (1.09 times more likely) of visiting a Wikipedia in their preferred language.

The primary link feature (in this context) was referencing rolling up the long list of languages by article count into a dropdown. As a side note, we typically call the language links that encircle the globe the 'primary language' links.

Were you referring to the new language by article count dropdown that is the subject of this talk page or the other links around the globe?

Hopefully this reply (and your lunch) helps? :)

I still don't get what did they receive... sorry. I though someone got "A" and someone got "B".... so are you describing something that both "A" and "B" show?

@Alexmar983 "dynamic primary links feature" refers to the links around the globe. These links are 'dynamic' because they change depending on your browsers language preference, for example: if your browser is set to Hungarian, the link to Hungarian wikipedia should be visible around the globe.

The point regarding this feature was meant to explain that in this A/B test, both the test group and the control group received this feature, but the test group (which was given the new layout) was more likely to use it.

now I got it

If some of the native speakers are having problems with these stats, the rest of us non-natives are going to have a much harder time following. 3.64% rise in engagement probability doesn't seem significant.

The portal gets 10-25 million views per day, so a 3.64% rise in engagement could mean an extra half-million visits into wikimedia projects. Perhaps we should include those kinds of numbers in the report, so people realize that even a fraction of a percent can make a pretty big difference.

Good point. Will try to remember to do so going forward.