View source for Growth/Analytics updates/Welcome survey editor activation rate

{{Wikimedia engineering project information
| name = Welcome survey editor activation rate
| description = The first published report using data from the "Personalized first day" project.
| start = 2018-11-19
| end = 2018-12-25
| group = {{ll|Growth|Growth}}
| lead = [[User:MWang (WMF)|Morten Warncke-Wang]]
| Phabricator =
| updates = 
| progress = 
| previous = 
| next = 
| projectpage = 
| backlog = 
| phabricator =
| display = {{{display|}}}
}}
{{tracked|T212799}}
{{tracked|T228117}}
== Summary ==
When we deployed the "[[Growth/Personalized first day/Welcome survey|welcome survey]]" to Czech and Korean Wikipedia, our main concern was that it would lead to fewer newly registered users becoming editors within the first 24 hours after registering (what we call "editor activation"). This concern is [[Growth/Analytics updates/Personalized first day experiment plan#Experiment plan|described in our experiment plan]], and is why we ran the survey as a randomized A/B test over the course of a month: half of new registrations received the survey immediately after creating their account, half did not get the survey at all and returned to the context they were in before creating their account. In this update, we describe the results of that A/B test, answering the question: does having the welcome survey affect editor activation rate? We find that there '''does not appear to be a statistically significant difference in activation rate''' between the survey and control groups in either of the two Wikipedias.

In testing Arabic Wikipedia in August 2019, we also find that there is no statistically significant difference in activation rate between the survey and control groups.

Next, the Growth team will test a more heavily designed version of the survey, [[Growth/Personalized first day#Design for Welcome Survey|Variation C]], against the simple Variation A tested in the experiment discussed on this page.
{{__TOC__}}
== Background ==
The survey was deployed to Czech and Korean Wikipedias on November 19, 2018, shortly after 19:00 UTC. In this analysis, we use data from deployment until December 25, so as to use whole weeks.<ref group="footnote">Wikipedia activity tends to fluctuate in weekly cycles, see for example Yasseri, Taha, Robert Sumi, and János Kertész. "[https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0030091 Circadian patterns of wikipedia editorial activity: A demographic analysis.]" ''PloS one'' 7.1 (2012): e30091.</ref> While we had one more week of data available at the time of analysis, we discarded it due to a [[phab:T212667|spambot attack]] affecting registrations on Korean Wikipedia.

In addition to limiting accounts by date of registration, we also apply several other filters:

* First, the survey is only shown to users who register on the given wiki, so we filter out users who already had accounts on another wiki (also known as "autocreated accounts").
* Secondly, we filter out accounts created through Wikipedia's API as those are mainly accounts created from the Wikipedia Android and iOS apps, and the survey is not running on either of those apps.
* Lastly, we remove known test accounts created by Growth Team members.

== Results ==
Our dataset contains {{Formatnum:1617}} accounts in Czech Wikipedia and {{Formatnum:2140}} accounts in Korean Wikipedia. For each of these accounts, we use Wikipedia's edit history to calculate whether a user made at least one edit within 24 hours after registration. We only calculate this for the first 24 hours because our previous analysis revealed that users who become editors are most likely making that transition quickly, only about 10% of those who ever make an edit make their first edit later than 24 hours after registration. With data on whether they edit, and whether they were shown the welcome survey or not, we can create 2x2 contingency matrices for both wikis. They are shown in Tables 1–4 below.
{|
|
{| class="wikitable"
|+Table 1: Activation counts by group, Czech
!Experiment group
!Did not edit
!Made ≥ 1 edit
!Total
|-
|Control
| style="text-align: right;" |{{formatnum:459}}
| style="text-align: right;" |{{formatnum:342}}
| style="text-align: right;" |{{formatnum:801}}
|-
|Survey
| style="text-align: right;" |{{formatnum:441}}
| style="text-align: right;" |{{formatnum:375}}
| style="text-align: right;" |{{formatnum:816}}
|-
|''Total''
| style="text-align: right;" |{{formatnum:900}}
| style="text-align: right;" |{{formatnum:717}}
| style="text-align: right;" |{{formatnum:1617}}
|}
|
{| class="wikitable"
|+Table 2: Activation % by group, Czech
!Experiment group
!Did not edit
!Made ≥ 1 edit
!Total
|-
|Control
| style="text-align: right;" |{{formatnum:57.3}}%
| style="text-align: right;" |{{formatnum:42.7}}%
| style="text-align: right;" |{{formatnum:100.0}}%
|-
|Survey
| style="text-align: right;" |{{formatnum:54.0}}%
| style="text-align: right;" |{{formatnum:46.0}}%
| style="text-align: right;" |{{formatnum:100.0}}%
|-
|''Total''
| style="text-align: right;" |{{formatnum:55.7}}%
| style="text-align: right;" |{{formatnum:44.3}}%
| style="text-align: right;" |{{formatnum:100.0}}%
|}
|-
|
{| class="wikitable"
|+Table 3: Activation counts by group, Korean
!Experiment group
!Did not edit
!Made ≥ 1 edit
!Total
|-
|Control
| style="text-align: right;" |{{formatnum:618}}
| style="text-align: right;" |{{formatnum:451}}
| style="text-align: right;" |{{formatnum:1069}}
|-
|Survey
| style="text-align: right;" |{{formatnum:658}}
| style="text-align: right;" |{{formatnum:413}}
| style="text-align: right;" |{{formatnum:1071}}
|-
|''Total''
| style="text-align: right;" |{{formatnum:1276}}
| style="text-align: right;" |{{formatnum:864}}
| style="text-align: right;" |{{formatnum:2140}}
|}
|
{| class="wikitable"
|+Table 4: Activation % by group, Korean
!Experiment group
!Did not edit
!Made ≥ 1 edit
!Total
|-
|Control
| style="text-align: right;" |{{formatnum:57.8}}%
| style="text-align: right;" |{{formatnum:42.2}}%
| style="text-align: right;" |{{formatnum:100.0}}%
|-
|Survey
| style="text-align: right;" |{{formatnum:61.4}}%
| style="text-align: right;" |{{formatnum:38.6}}%
| style="text-align: right;" |{{formatnum:100.0}}%
|-
|''Total''
| style="text-align: right;" |{{formatnum:59.6}}%
| style="text-align: right;" |{{formatnum:40.4}}%
| style="text-align: right;" |{{formatnum:100.0}}%
|}
|}
Note: in Tables 2 & 4, proportions are calculated ''per row''. This is to make it easier to compare activation rates for the survey and control groups.

The proportions shown in Tables 2 & 4 suggests conflicting trends between the survey and control groups in the two wikis. In Czech, the survey group has a slightly higher activation proportion (by 3.3pp), while in Korean it is slightly lower (by 3.6pp). But, are any of these differences statistically significant?

The answer is "no". Using a two-sample test of equality in proportions we find that neither the difference in Czech (<math>\Chi^2=1.61, df=1, p=0.20</math>) nor Korean (<math>\Chi^2=2.77, df=1, p=0.095</math>) is statistically significant.

== Addendum: Arabic Wikipedia ==
We deployed the Welcome Survey to Arabic Wikipedia on July 15, 2019, using the same A/B test configuration we used for Czech and Korean. Users who signed up on Arabic Wikipedia would be randomly assigned to either a survey or control group with 50% probability. In our analysis of the results we used five whole weeks of data, meaning all registrations up until August 26, 2019. As we did for Czech and Korean, auto-created accounts are removed from our dataset because the survey is only shown to local registrations. We also remove accounts created through the API (these are primarily app accounts), and known test accounts.

Our dataset contains {{formatnum:9524}} accounts, of which {{formatnum:4808}} ({{formatnum:50.5}}%) were in the control group, and {{formatnum:4716}} ({{formatnum:49.5}}%) were in the survey group. Similarly as for Czech and Korean Wikipedia, we use the MediaWiki databases to count how many edits these users made in the first 24 hours after registration. From this data we can then create 2x2 contingency tables, seen in Tables 5 & 6 below.
{|
|+
|
{| class="wikitable"
|+Table 5: Activation counts by group, Arabic
!Experiment group
!Did not edit
!Made ≥ 1 edit 
!Total
|-
|Control
| style="text-align: right;" |{{formatnum:3469}}
| style="text-align: right;" |{{formatnum:1339}}
| style="text-align: right;" |{{formatnum:4808}}
|-
|Survey
| style="text-align: right;" |{{formatnum:3372}}
| style="text-align: right;" |{{formatnum:1344}}
| style="text-align: right;" |{{formatnum:4716}}
|-
|''Total''
| style="text-align: right;" |{{formatnum:6841}}
| style="text-align: right;" |{{formatnum:2683}}
| style="text-align: right;" |{{formatnum:9525}}
|}
|
{| class="wikitable"
|+Table 6: Activation % by group, Arabic
!Experiment group
!Did not edit
!Made ≥ 1 edit
!Total
|-
|Control
| style="text-align: right;" |{{formatnum:72.2}}%
| style="text-align: right;" |{{formatnum:27.8}}%
| style="text-align: right;" |{{formatnum:100.0}}%
|-
|Survey
| style="text-align: right;" |{{formatnum:71.5}}%
| style="text-align: right;" |{{formatnum:28.5}}%
| style="text-align: right;" |{{formatnum:100.0}}%
|-
|''Total''
| style="text-align: right;" |{{formatnum:71.8}}%
| style="text-align: right;" |{{formatnum:28.2}}%
| style="text-align: right;" |{{formatnum:100.0}}%
|}
|}
Note: in Table 5 proportions are calculated ''per row''. This is to make it easier to compare activation rates for the survey and control groups.

There is a small difference in activation between the control and the survey group, but this difference is not statistically significant (<math>\Chi^2=0.47, df=1, p=0.50</math>). Therefore, we conclude that the welcome survey does not negatively impact whether newcomers stay on the wiki and make contributions.

== Footnotes ==
<references group="footnote" />