Growth/Analytics updates/Help Panel editor activation rate

Group:	Growth
Start:	2019-01-11
End:	2019-03-29
Lead:	Morten Warncke-Wang

Tracked in Phabricator
Task T206720

Summary

The overarching goal of the Help Panel is to provide users with easier ways to get help without them having to leave the editing context. For newly registered users this could mean they are more likely to make their first edit, or make that first edit sooner, which in our case means a potential increase in "editor activation" (the proportion of new users who make at least one edit within 24 hours after registration).

Analyzing the Help Panel's impact on editor activation is part of our experiment plan. In that plan we describe how we run an A/B test during which half of new registrations on our target wikis (Czech, Korean, and Vietnamese) get the Help Panel turned on by default, and half do not (whether they do or not is chosen at random when they register). Note that auto-created accounts (users who register but already have an account on another wiki) are not part of this experiment.

In this update, we describe the results from analyzing 2.5 months of data from the Czech and Korean Wikipedias and 1 month of data from Vietnamese Wikipedia. The conclusion of our analysis is that there does not appear to be a statistically significant difference in activation rate between the Help Panel and control groups in either of our target Wikipedias.

Next, the Growth Team is working on using statistical methods that allow us to analyze data from all three wikis together in order to increase our statistical power, as well as also looking at whether the Help Panel affects user activity. We're also working on determining future changes to the Help Panel to improve its functionality. Lastly, our A/B test will continue to run and gather data, so expect to see further analyses published in the future.

Background

The Help Panel was deployed to the Czech and Korean Wikipedias on January 11, 2019, and to the Vietnamese Wikipedia on February 28, 2019. In this analysis, we use data from deployment until March 29, 2019.

In addition to limiting accounts by date of registration, we also apply several other filters:

First, the Help Panel is only activated for users who register on the given wiki, so we filter out users who already had accounts on other wikis (also known as "autocreated accounts".
Secondly, we filter out accounts created through Wikipedia's API as those are mainly accounts created from the Wikipedia Android and iOS apps and the Help Panel is not available on either of the apps.
Third, we remove known test accounts created by members of the Growth Team.
Lastly, we remove users who turned the Help Panel preference on or off as this type of self-selection into or out of the Help Panel group conflicts with the experiment's random assignment. In practice, we find that only a handful of users changed this preference.

The result is a dataset that contains 3,848 accounts in Czech Wikipedia, 3,690 accounts in Korean Wikipedia, and 5,283 accounts in Vietnamese Wikipedia.

Power analysis

Before analyzing the results, we estimated what effect sizes we would be able to detect based on the number of users in our dataset for each of the three wikis. In this analysis, we used historical data to calculate an estimate of what the average activation rate for each wiki has been. The results are shown in Table 1 below.

Table 1: Historic activation rate and effect thresholds
Wiki	Activation rate	Effect size	Lower bound	Upper bound
Czech	37.3%	±13%	32.4%	42.2%
Korean	38.5%	±12%	33.8%	43.2%
Vietnamese	15.7%	−18%, +19%	12.8%	18.7%

In Table 1, the "activation rate" column is the estimated rate based on historical data, and can be interpreted as what we would expect the activation rate of the control group to be. The "effect size" column shows what the change in percent (not percentage points) needs to be for us to be able to detect it. The lower and upper bound columns are then calculated based on the historical activation rate and the effect size. Note that if the actual activation rate is different from the estimate, then the three other columns all change due to how the test of difference in proportions behaves.

Another way to read Table 1 is that for the Czech Wikipedia we would expect the activation rate of the control group to be 37.3%, and if the activation rate of the Help Panel group is either below 32.4% or above 42.2% that would mean a statistically significant result.

Results

For each of the accounts in our dataset, we use Wikipedia's edit history to calculate whether a user made at least one edit within 24 hours after registration. We only calculate this for the first 24 hours because our previous analysis revealed that users who become editors are most likely making that transition quickly, only about 10% of those who ever make an edit make their first edit later than 24 hours after registration. With data on whether they edit and whether they were part of the Help Panel or control group we can create 2x2 contingency matrices for each of the three wikis. They are shown in Tables 2–7 below.

Table 2: Activation counts by group, Czech
Experiment group	Did not edit	Made ≥ 1 edit	Total
Control	1,137	777	1,914
Help Panel	1,174	760	1,934
Total	2,311	1,537	3,848

Table 3: Activation % by group, Czech
Experiment group	Did not edit	Made ≥ 1 edit	Total
Control	59.4%	40.6%	100.0%
Help Panel	60.7%	39.3%	100.0%
Total	60.1%	39.9%	100.0%

Table 4: Activation counts by group, Korean
Experiment group	Did not edit	Made ≥ 1 edit	Total
Control	1,065	737	1,802
Help Panel	1,118	770	1,888
Total	1,802	1,888	3,690

Table 5: Activation % by group, Korean
Experiment group	Did not edit	Made ≥ 1 edit	Total
Control	59.1%	40.9%	100.0%
Help Panel	59.2%	40.8%	100.0%
Total	59.2%	40.8%	100.0%

Table 6: Activation counts by group, Vietnamese
Experiment group	Did not edit	Made ≥ 1 edit	Total
Control	2,298	352	2,650
Help Panel	2,284	349	2,633
Total	4,582	701	5,283

Table 7: Activation % by group, Vietnamese
Experiment group	Did not edit	Made ≥ 1 edit	Total
Control	86.7%	13.3%	100.0%
Help Panel	86.7%	13.3%	100.0%
Total	86.7%	13.3%	100.0%

Note: in Tables 3, 5, and 7, proportions are calculated per row. This is to make it easier to compare activation rates between the Help Panel and control groups.

The proportions shown in Tables 3, 5, and 7 suggests that if there is any difference in activation rate between the Help Panel and control groups it is likely small, and would therefore require a longer experiment (or data from more wikis) to be detectable. Using a two-sample test of equality in proportions we find no statistical significance for any of the three wikis in our dataset.