Talk:New Developers/Quarterly/2017-10

Wikimania Hackathon, GSoC/Outreachy, African developers group...

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

We are getting closer to a clear idea about the survey related information in this report. How do you envision the mention of other activities related to new developers? The Wikimania Hackathon, GSoC/Outreachy, the African developers group... what else? And who writes what?

For the next edition of this report, I think it is worth having the skeleton available as soon as possible, in order to add these topics as they come (it is easy to forget things a couple months later) and even commission the writing to someone related, who wouldn't need to wait until the end of the quarter to write their part. Qgil-WMF (talk) 07:31, 11 September 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Subscriptions though the Newsletter extension

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

What about creating a newsletter for this report, allowing people to subscribe easily? This doesn't need to wait to the release of the first report. Qgil-WMF (talk) 07:33, 11 September 2017 (UTC)

How about the following title, description and URL?

Title: New Developers Quarterly 

Description: Updates from new developers activities in the Wikimedia movement, including metrics, survey analysis, key findings, and lessons learned

URL for main page: New Developers User:SSethi_(WMF) 21:35, 6 October 2017 (UTC)

Very good! Done. Subscribe!

Newsletter:New Developers Quarterly Qgil-WMF (talk) 07:56, 9 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Doubts

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Srishti asks:

Time to get a Bitergia overview with a specific focus on topics mentioned below:

General

• For the new developers, how do we filter out the contributions made to third party repositories (as in T146135#3176718)?
  • Go to C_Gerrit_Demo and copy the second block from phab:T146135#3176718 into the search bar. The results will not change much though, this is just a safety measure. And Andre should probably check the entries in that manual blacklist again.
• Pulling email addresses.. Is it possible that Srishti understands this process as well? Or Srishti emails Andre 2-3 times in the month of September asking for more emails?   
  • Andre to ask Bitergia a week before September ends for updated and complete quarter data, as the data on C_Gerrit_Demo is not yet automatically updated.

About specific metrics

• Number of commits we received and merged from new developers in the last quarter - how do we pull proposed commits that landed in Gerrit and the ones that got merged/landed in git from new developers  
  • There is no way to do this on C_Gerrit_Demo. The only workaround is taking all the names of new authors from the "new Authors" list on C_Gerrit_Demo (CSV export) and constructing a query for the search field on https://wikimedia.biterg.io/app/kibana#/dashboard/Gerrit by entering author_name:"foo" OR author_name:"bar". Then use the "Status" circle in the middle by hovering your mouse pointer over it and take the "Count" numbers for NEW, ABANDONED, MERGED from there.
• Number of new developers who actively contributed in the last quarter - (More than one commitset / patchset / contribs) etc. What is “contribs” in the "New Authors" widget on the C_Gerrit_Demo page?
  • contribs = changesets in Gerrit. Not patchsets within one changeset.
• New developers active 1 year after their first contribution - How to calculate this?
  • By following the complicated steps on https://phabricator.wikimedia.org/T160430#3383647 User:SSethi_(WMF) 19:21, 28 September 2017 (UTC)

@AKlapper (WMF) First, some minor comments:

• Some of the links in the metrics table take you to the absolute time range. For example for the metric, new developers attracted, the data shown is from a date in the last week of June to now. See https://wikimedia.biterg.io/goto/1482e35f5eda39153c95bc485b5ac74a. In case our calculations don’t match then that might be the reason: I’m pulling data via the relative time range between July 1st 00:00:00.000 to now.
• Tiny picks: “Filter data of independent contributors in Gerrit..” links in https://phabricator.wikimedia.org/T160430#3383647 don’t work. Though now, I know, how to get there still, they require an update. I get an error: short url must be in the target. Also, the first link should be “Filter data … in C_Gerrit” and not Gerrit?

Questions

• Does the graph with title “Trends in onboarding and retention per quarter between July 2015 - September 2017” make any sense? Do you think the data I’ve calculated is right? I want to highlight the trend, the big picture, by showing the onboarding and retention stats together, but I’m now wondering if it might be better to present the onboarding and retention trend separately, also because of the extremely opposite figures in there. Also, I see an issue which is that for recent quarters we will not be able to show retention numbers. See “NM” on the table. Do you have any thoughts or you see an alternative to show the overall trend in a better way?   
• Outreach programs GSoC / Outreachy metrics - Inviting you to a spreadsheet where I’ve pulled information of the students from previous year with their last code activity. Help me understand how retention should be calculated in this case :)
• How do you generate GOTO links? User:SSethi_(WMF) 06:36, 29 September 2017 (UTC)

(Please let's split the many discussions in topics on their own. Discussing like this is prone to fail.) Qgil-WMF (talk) 08:17, 29 September 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Measuring retention

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@AKlapper (WMF) asks:

Under "Developer outreach programs and events > Number of new developers onboarded and retained at: > Wikimedia Hackathon 2017 / Onboarded and retained", how is 'retained' defined exactly? Check for any activity between 20170701 and 20170930 after Hackathon in 201705?

• @SSethi (WMF): Yes, check for any activity at all between now and the hackathon (20170701 and 20170930) Qgil-WMF (talk) 08:09, 29 September 2017 (UTC)

Cool graphic! Thank you @SSethi (WMF). About the retention data, I think we need to apply the same criteria as in m:Technical_Collaboration/Metrics#Onboarding_New_Developers ("New volunteers active one year (±3months) after their first contribution"). Just copy the numbers that Andre is using there?

@AKlapper (WMF) have we documented somewhere the exact definition of our KPIs and how we are measuring them? Qgil-WMF (talk) 08:14, 29 September 2017 (UTC)

For generating graphs, what help I would need now would be to understand what QoQ and YoY means, and some more information about the long-term trend graph in  m:Technical_Collaboration/Metrics#Onboarding_New_Developers. I am considering to make a similar graph, might change size, type and add some information along X & Y axis. User:SSethi_(WMF) 19:14, 29 September 2017 (UTC)

@SSethi (WMF): Comparing Quarter on Quarter (which translates to "This last quarter that just ended vs the previous quarter") and Year on Year (which translates to "This last quarter that just ended vs the same quarter in the previous calendar year", at least in my interpretation, and not "value for the full last 12 months vs value of the 12-24months ago).

Which "some more information" may I provide, please? :) AKlapper (WMF) (talk) 22:34, 29 September 2017 (UTC)

@AKlapper (WMF) No, we don't need any more information right now on this! :) It was unclear to me until I opened the source of the Metrics page and looked at the comments written along with the values of Sparkline graph. See now the two big graphs in the "Trends in attraction and retention between mm 2016 – mm 2017." It is using the same data we have in m:Technical_Collaboration/Metrics#Onboarding_New_Developers. Let me know whenever you are done updating the Metrics, and then I will use them in the graphs. One question: the Y values used to draw the Sparkline graph for the third KPI (New volunteers active one year (±3months) after their first contribution) are too big (e.g., y1 1633, y2 690 and so on) or see the number of developers axis in the second big graph representing retention rates in the report. I am wondering why is that so, and for the purpose of showing the values on the graph, could we get small values? User:SSethi_(WMF) 07:45, 2 October 2017 (UTC)

@SSethi (WMF): Re: "too big numbers": See the comments in the wiki text source on https://meta.wikimedia.org/w/index.php?title=Technical_Collaboration/Metrics:

x1 = 1459468799 | y1 = 1633| <!-- 2016Q1: ≙ 16.33% = 8 out of 49 -->
x2 = 1467331199 | y2 = 690| <!-- 2016Q2: ≙ 6.90% = 2 out of 29 -->

Sparkline does not accept non-integer numbers as input. Feel free to cut/round. AKlapper (WMF) (talk) 10:36, 2 October 2017 (UTC)

@Qgil-WMF: I'm not sure I get the question "have we documented somewhere the exact definition of our KPIs and how we are measuring them?". Could you clarify what's missing to being 'exact' when looking at https://meta.wikimedia.org/wiki/Technical_Collaboration/Metrics#Onboarding_New_Developers ? Because if something is unclear I would love to clarify already on the meta: page. (Discussion took place in https://phabricator.wikimedia.org/T160430 but you don't want to go through that again, trust me.) AKlapper (WMF) (talk) 22:32, 29 September 2017 (UTC)

Nothing to add.  :) Qgil-WMF (talk) 12:11, 4 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Volunteers vs professionals

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@AKlapper (WMF) asks:

• All links above include any new developers, also staff, as the phrasing does not imply 'volunteers'. Should I changes those queries to only include author_org_name:"Independent" OR author_org_name:"Unknown"?
  • @SSethi (WMF): For a moment, I was about to say that it makes total sense to me and that exclude the staffers. But, I think if there are staffers new to Wikimedia code contribution process and trying to contribute in their volunteer time only, it would be good to include them in this study as well, unless you have a different opinion. Qgil-WMF (talk) 08:16, 29 September 2017 (UTC)

Professional developers contributing from their professional usernames / addresses should not be counted. From a volunteer developer outreach perspective, we aren't doing anything to reach out to them, onboarding them, or retaining them. If someone wants to contribute purely as a volunteer during or after their contract (and be counted as volunteer), they can do so using a personal username / email address.

Besides, mixing professional identities with volunteer metrics is prone to mistakes. The principle of no professional identities in volunteer metrics is simpler and more reliable. Qgil-WMF (talk) 08:22, 29 September 2017 (UTC)

We cannot easily differentiate staffers trying to contribute in their volunteer time only - see https://phabricator.wikimedia.org/T95238. So unaffiliated only that means. @SSethi (WMF): Should probably be explicitly mentioned that those stats are supposed to be about //volunteer// developers? AKlapper (WMF) (talk) 22:23, 29 September 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Pulling and copying data

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I see that @AKlapper (WMF) and @SSethi (WMF) are both pulling data. While in an ideal world we should have robust definitions and methods that allow anyone to pull data obtaining the same results, I think we are not there yet. I would feel better if we keep the roles assigned, Andre takes care of quantitative data, and Srishti takes care of qualitative data (the survey) and analysis. Otherwise we are going to spend a lot of extra time trying to retrieve the correct data and verifying it. Qgil-WMF (talk) 08:30, 29 September 2017 (UTC)

At some point, I would like to create a subpage of the New Dev report and add information about how we pull data for all the metrics we are including in the report (with help from Andre). User:SSethi_(WMF) 19:14, 29 September 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Our three KPIs, at the top

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

We put a lot of thought in our KPIs and, well, they are Key. Therefore, I think that all three should be at the top with fully developed graphs that allow to check the values.

m:Technical_Collaboration/Metrics#Onboarding_New_Developers

These three graphics should set the tone for the report. Do we have more volunteers in total? Did we get more new volunteers last quarter? Have we improved our retention levels in the last quarter? The mood for the rest of the report will be very different if the answer to these three questions is "YES!!!" or "Not at all..." Qgil-WMF (talk) 13:57, 29 September 2017 (UTC)

Sounds good to me! I will remove the giant graph which is in there right now on the top and then bring the first three metrics in the table which are our top three KPIs at the top of the "New Developers Metrics and Trends" section and highlight them more. Yes, I will the pull final updated data from our metrics page and use that to generate graphs. User:SSethi_(WMF) 19:15, 29 September 2017 (UTC)

@AKlapper (WMF) can you add QoQ and YoY to the KPIs, please? Qgil-WMF (talk) 10:20, 4 October 2017 (UTC)

Added QoQ/YoY items to the 3 KPIs by copying from meta. (Data might slightly change tomorrow as I need to re-calculate some stuff as DB issues got... 'more complicated'.)

(Unrelated: Going to take the mw: layout and use the same on meta: for the KPIs. Too much work to update two pages with different table designs instead of copy&paste.)

Assuming this thread is resolved. AKlapper (WMF) (talk) 20:34, 4 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

"Onboarded" vs "attracted"

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I'd say we definitely "attracted" folks if they contributed but did not always necessarily also "onboard" them? Might be just my understanding of English (and/or Marketing) language though. AKlapper (WMF) (talk) 22:23, 29 September 2017 (UTC)

Yes! Made this change. User:SSethi_(WMF) 07:28, 2 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Nitpicking on formatting of text

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Both - (long) and – (even longer) are used inconsistently for list items (like "1 - bar; 2 - foo"). "#1" or "1." might be more common anyway?

I'd also think that using a colon for name:value pairs ("Moo: 43; Faa: 21") is most common?

I'm just a bit confused by seeing many many hyphens with different meanings. AKlapper (WMF) (talk) 10:27, 2 October 2017 (UTC)

I am guessing not any more? Resolving for now! User:SSethi_(WMF) 07:04, 6 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Single-option vs multiple options?

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Some survey answers allowed choosing multiple options I assume (like "fixing bugs") while others only choosing one option. Should those that offered multiple options explicitly mention that, for clarity that you'd get >100% when summing up? (I don't know, just a thought.) AKlapper (WMF) (talk) 10:29, 2 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

First draft of key findings and timeline

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@AKlapper (WMF) @Qgil-WMF I can iterate on the language, but first do the findings in there make sense? Also, the timeline?

I'm first trying to finish the remaining bits and pieces of the report and then will move on to do the content organization (such as that of survey analysis), and fix nitpicks as we have already discussed.  User:SSethi_(WMF) 08:55, 3 October 2017 (UTC)

Key findings, the overall impression is very good! I think there is some nuance of course, but we can resolve that here and with edits.

1. Regular trends in attraction, and stagnating in retention of new developers

Good point, but since we are saying that those numbers are relatively stable, I would put them in the title. However, we also need to specify how we are counting that average. "Four last quarters" seems a reasonable way to offset seasonal movements and to still provide recent data. If we count the last four quarters, then the numbers I get are 54.5 developers and 5.16% retention. Also, one has to wonder why the retention percentages in 2016 are double/triple. Something to mention and look at in the future?

2. New Developers tend to become involved in contributing code to Wikimedia software projects when they learn about it through their peers

Good. But we need to say that this is preliminary data based on a few responses, to be checked in future quarters/

3. New developers tend to be working professionals with over five years of experience developing software

Interesting but see above about preliminary data. Now it sounds as a solid data point.

4. Wikimedia techinical community seems to be active in reviewing changesets of new developers

The wording can be more something like "code review, less of a problem for newcomers than we thought?"

5. Three projects that received most contributions from new developers in the previous quarter are Extension:Example, Wikimedia Commons Android app, and MediaWiki Core

As far as I can see, this is about the Wikipedia app for Android, not the Commons app.

6. New developers struggle with the Wikimedia code review process

OK  

7. Dedicated spaces, platforms, and learning environments for new developers both online and offline empower them and contribute to an engaging experience

OK 

8. Featuring newcomer-friendly projects helps in attracting developers

OK

9. Continued guidance offered by the local community, support in easy onboarding of new developers

OK, but maybe worth to promote this one a bit higher in the list, with a bit more details? Qgil-WMF (talk) 12:10, 4 October 2017 (UTC)

@SSethi (WMF): Re #5: Also note that Extension:Example is only popular for contributions because people find it via Gerrit/Tutorial#Download code using Git and use it as a playground to try Gerrit //in general//. Hence nearly all proposed patches are not actual 'functional' code changes. I can tell because I watch that repository to manually add "Welcome!" replies as long as phab:T73357 is unresolved. AKlapper (WMF) (talk) 21:39, 4 October 2017 (UTC)

Made changes accordingly! thank you! User:SSethi_(WMF) 07:01, 6 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Event newcomers tracking

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I've taken another look at the newcomer spreadsheet from Vienna/ Montreal and marked a few as "Yes". I guess at this point the ones marked with neither "Yes/ No" are also people that we would want to track. @AKlapper (WMF) Is there any manual work that I can do to help reduce your workload at this point of pulling stats? User:SSethi_(WMF) 18:23, 3 October 2017 (UTC)

I don't think you can if you don't want to import a huge JSON database dumpfile (that you cannot access by default) via a special tool into your local MariaDB database to run this query, but thanks a lot for the offer... :D AKlapper (WMF) (talk) 19:12, 3 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Review of changesets by new developers

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I like this graph but I see two problems:

1. It should specify "as of YYYY-MM-DD" because those number will change (hopefully) between the day the data is taken and the date someone looks at the graph.
2. If we count July-September and then we capture the data on October 1st, then it is very normal to have a high number of changesets open without review. Should we wait an extra week and then capture the data? One week to process a potential changeset uploaded by a newcomer on the last day of the quarter seems to be a good enough margin.

@AKlapper (WMF) Qgil-WMF (talk) 10:27, 4 October 2017 (UTC)

Good point, thanks. Per definition that is always an issue with any retention graph: The closer you get to "now" the larger the number of unreviewed patches is will be. See for example the two graphs at the top of https://wikimedia.biterg.io/app/kibana#/dashboard/Gerrit-Backlog.

While "a week" (or any other timeframe) feels arbitrary, given the issues I already have with getting reliable data I concur for the sake of practicability (means: data quality).

And yes I will add an "as of". AKlapper (WMF) (talk) 11:17, 4 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

New developers who contributed more than one changeset

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I'm not very convinced about the usefulness of this data point? What about New developers by number of contributions? 1, 2-5, 5-10, more than 10?

Even this might be a bit fishy in the context of one quarter because someone joining in July will have a lot more time than someone joining in September, and the later will not count as newcomer anymore in the data for the following quarter.

@AKlapper (WMF) Qgil-WMF (talk) 10:34, 4 October 2017 (UTC)

I am not convinced either so I am going to drop this entirely. In the end it is another variation of "was their some retention" which we already measure with a different time diff ("12-15 months" instead of "in the same quarter"). AKlapper (WMF) (talk) 10:46, 4 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Projects with most received contributions by new developers

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I cannot map the numbers shown in the graph with the numbers seen in the page following the link. However, the graph seems to be counting number of contributions. Wouldn't it be more useful to count number of developers?

FICTIONAL EXAMPLE: If 2 developers contribute 10 patches each to Wikipedia Android app and 10 developers contribute 2 patches each to MediaWiki Core, both make a total of 20 patches, but it is a lot more interesting to see that MediaWiki Core got 10 newcomers.

Focusing on people rather than patches also diminishes the problem of comparing patches by complexity. Instead, the complexity of attracting a new developer is quite comparable across people and projects.

@AKlapper (WMF) Qgil-WMF (talk) 10:40, 4 October 2017 (UTC)

@Qgil-WMF Uh, thanks. Indeed 25 instead of 28 repositories. That's because excluding staff not marked as volunteers is... still WIP as I ran into new puzzling data corruption issues yesterday night (I already reported them to Bitergia).

The graph counts what you expect it to count: 51 developers contributed to 25 repos. It does not count the number of contributions. The number of contributions is shown in the "New Authors" list widget. If you click on the 3rd biggest item in the pie chart (apps/android/wikipedia with a value of 3), you will see 3 authors listed under "New Authors". With in total 24 contributions. (That the list of "new Authors" still includes staff is a different issue I still need to solve, see my second sentence in this comment.)

I have adjusted the legend under User:SSethi (WMF)/New Developers Quarterly Report (July - September 2017)#Projects with most received contributions by new developers

So I guess this is resolved. AKlapper (WMF) (talk) 11:05, 4 October 2017 (UTC)

I still find the title confusing. Why not simply "Projects with most new volunteers"? Qgil-WMF (talk) 12:42, 18 October 2017 (UTC)

Yeah, good point. Done. AKlapper (WMF) (talk) 16:02, 18 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

First draft of recommendation for next steps

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I've added some recommendation for next steps. They might be a bit biased because while framing them I might have been thinking of tasks that are already out there on Phabricator that we have plans of working on in the upcoming quarters, or might be only covering stuff I know more or associated closely with, or we have vaguely discussed them here and there. Do they make sense to you?  User:SSethi_(WMF) 08:11, 5 October 2017 (UTC)

Hm, I wonder whether we are venturing too much into conclusions here. We are taking some next steps because we think is the right thing to do, but here we should focus on next steps clearly related to findings in the report. For instance "Consider getting Wikipedia Android Apps project listed as a featured project" is a humble step, but a very logical and practical one. If there are points that have a weak relation, then I would remove them.

Then, I think the text could be shorter, the style simpler. For instance, if the problem was that mapping code contributions to hackathon participants may be complicated and we need to find a way to connect both better in the registration form, then we can simply say that.

A next step that we have discussed and agreed is a shorter survey with the intention to obtain a higher percentage of responses. Qgil-WMF (talk) 10:54, 5 October 2017 (UTC)

Made changes accordingly! Totally made sense to take the points out that were not necessarily related with the findings from the report. User:SSethi_(WMF) 07:01, 6 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Data backend issues

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@Qgil-WMF, @SSethi (WMF): Regarding User:SSethi (WMF)/New Developers Quarterly Report (July - September 2017)#Review of changesets by new developers and the section after (the two pie diagrams): Note that I do not fully trust the data as it is inconsistent in wikimedia.biterg.io itself: https://phabricator.wikimedia.org/T177569

Should that explicitly be mentioned in a footnote? AKlapper (WMF) (talk) 02:46, 6 October 2017 (UTC)

If you don't trust the data, I would remove it and publish it in future reports when we can trust it. Qgil-WMF (talk) 11:29, 6 October 2017 (UTC)

Hmm. As this is about "Number of new developers in Gerrit" that would also invalidate KPI2 and to smaller extent KPI3 on https://meta.wikimedia.org/wiki/Technical_Collaboration/Metrics#Onboarding_New_Developers . I believe that the ballpark figure is fine, it' just that we might be of by max 15% for the total number. (Do we have a strict deadline? Crossing fingers that Bitergia investigates/clarifies soon...) AKlapper (WMF) (talk) 11:49, 6 October 2017 (UTC)

The strict deadline is next Wednesday, October 11. The QCI presentation is on the 12th. Qgil-WMF (talk) 13:20, 6 October 2017 (UTC)

We're going to keep those numbers as the error span feels not neglectable but acceptable. AKlapper (WMF) (talk) 21:41, 10 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Where should this report ultimately live?

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@AKlapper (WMF) @Qgil-WMF Perhaps in a day or two time to send it away from my home to elsewhere.. but where Meta/ MediaWiki? Also, how about the title, it be the same? User:SSethi_(WMF) 07:14, 6 October 2017 (UTC)

[[New Developers/Report]] for the main page and then /October for this edition?

"New Developers Quarterly Report" sounds good to me. If we remove "quarterly" then it is shorter, but at the same time might give the impression that is a one-time report.

Please do not send it before the three of us are happy with it!  :) Qgil-WMF (talk) 13:27, 6 October 2017 (UTC)

Not sending it for right now, making sure, it will then be [[New Developers/Quarterly Report/October]]?? User:SSethi_(WMF) 21:16, 6 October 2017 (UTC)

Based on the name you proposed for the new newsletter (which I like a lot), the location could be

New Developers/Quarterly/2017-10

"YYYY-MM" is a unique identifier (I had missed this little detail).  :) Starting with the year will assure an automatic chronological classification in lists. Qgil-WMF (talk) 08:04, 9 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Qualtrics & tracking when a link is clicked

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Some info to share -- I called Qualtrics customer care today to inquire if there was a possibility to find out whether an email was opened or a link in the email body was accessed. I found that the plan we are currently on does not provide us access to this information. There might be alternatives to do so, for example, this link tells how to track when an external link is clicked: https://stackoverflow.com/questions/38225021/tracking-when-an-external-link-is-clicked-in-qualtrics-with-javascript

Though we have plans to iterate on the survey, it would be good to keep in mind that completion rate for the current survey was 69% (13 people started filling out the survey, and 9 finished it). So at this point, we will not be able to tell, if people opened the email containing the link to the survey or clicked through it.   User:SSethi_(WMF) 07:54, 6 October 2017 (UTC)

Which underlying problem would this solve which is bigger than my generic privacy concerns? :) How and why does it matter to know this?

(I'd personally always recommend to block having emails automatically load online content (like web beacons) or executing embedded JavaScript.) AKlapper (WMF) (talk) 09:33, 6 October 2017 (UTC)

Yeah, I think we had discussed this somewhere, and we decided that it was not relevant to know who had opened emails or not. What matters is the % of people that submit their answers to the survey. Qgil-WMF (talk) 13:22, 6 October 2017 (UTC)

Yeah, agree, just that the low response rate is due to poor survey design or people not interested in our emails, there would be no way to identify that. User:SSethi_(WMF) 21:14, 6 October 2017 (UTC)

The difference between the theoretical scenarios "not interested, didn't click" and "not interested, did click" is rather fuzzy though. Let's focus on simplifying the survey, and then let's review the situation after obtaining new participation data. Qgil-WMF (talk) 08:06, 9 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Confusing percentage values for multiselect options

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@SSethi (WMF) I mentioned this briefly but wanted to log it: Looking at User:SSethi (WMF)/New Developers Quarterly Report (July - September 2017)#Contributions :

As this is a multiselect option, you and I know that 9 developers checked a total of 31 checkboxes, so you calculate 100/31*6. But nobody else knows that number '31'. Hence I only know that 9 people replied in total (=100%), and 6 people out of those 9 said they 'fix bugs'. And that is not 19.35% but 66.67%? AKlapper (WMF) (talk) 09:28, 6 October 2017 (UTC)

I had added this text earlier "Participants could select more than one option," I thought it was sufficient but may be not?  User:SSethi_(WMF) 21:19, 6 October 2017 (UTC)

@ SSethi (WMF) If no readers (apart from you and me) know that the magic number is 31 this chart might become one of the greatest unresolved mysteries of... ah well, if 6 out of 9 people say "Yes I do X" to answer X, that answer X got 66.67%. Regardless of some hidden dependency on the popularity of saying "Yes I also do Y" to answer Y. Multi-option in my understanding means that the sum must be ≥100%. AKlapper (WMF) (talk) 14:05, 7 October 2017 (UTC)

I agree that this is confusing. For instance, "35.48% of respondents of this survey said they fix or report bugs"... is this really true? They could also be 19%, with people clicking both options. I would remove percentages altogether. They don't tell much anyway. Qgil-WMF (talk) 15:17, 10 October 2017 (UTC)

I would also remove percentages in "Challenges" and "Experience...". When the total universe is 9 people, talking about "55.65%" or "33.33%" feels odd. "Motivations" keeps the points in a more generic way, and that sounds more frank and humble. Qgil-WMF (talk) 15:20, 10 October 2017 (UTC)

Added an additional column "Count" in "Types of Contributions" that helps explain why the total count of responses=31 when n=9. Removed percentages in "Challenges" and "Experience". User:SSethi_(WMF) 18:08, 10 October 2017 (UTC)

I have removed the percentages altogether. They provide a "scientific flair" that in this case we cannot sustain. Qgil-WMF (talk) 07:03, 11 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Timeline

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@SSethi (WMF) I thought the timeline starting from most recent wouldn't bother me, but somehow it does. :D Is there any reason not to follow a chronological sort? Qgil-WMF (talk) 14:38, 10 October 2017 (UTC)

No strong feelings, so just changed the order. User:SSethi_(WMF) 18:10, 10 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Review of changesets by new volunteers

We don't need to fix here and now, but in "Review of changesets by new volunteers" it would be useful to offer a link to the patches waiting for decision. Maybe we get reviewers willing to help.

1. Qgil-WMF (talk) 15:10, 10 October 2017 (UTC)

"indicating that the completion rate of the survey was 69%"

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I would remove "indicating that the completion rate of the survey was 69%." because in a way it is not very useful, and in a way might be misleading. Qgil-WMF (talk) 15:12, 10 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Phragile?

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

Have been new developers exposed to Phragile? Qgil-WMF (talk) 15:21, 10 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

"Autonomy for development teams"?

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

"More autonomy for development teams: build a trust-based DevOps culture allowing teams to adopt Continuous Integration/Continuous Delivery practices and deliver value in small iterations (current deployment guide mentions a lead time of over six weeks for new services!)."

Does this sound like something that a new volunteer would say? Not a new professional developer hired?

This makes me think. Was the list of participants in the survey vetted, in order to assure that they were volunteers only, not new hires? Qgil-WMF (talk) 15:23, 10 October 2017 (UTC)

The comment below about Phragile in the topic below is also from the same individual response. This response is from a developer with a wikimedia.de address.. so yeah I sent the emails to everybody then from the list I had. Six people from July and August, who received the survey were staff members. Ahha, hope not to repeat the same mistake next time, but now to make up for this, should I just add a point in the Key findings as the next step towards improvement or something else?  User:SSethi_(WMF) 20:39, 10 October 2017 (UTC)

Right. No problem, another lesson learned. We need to have a documented sequence of steps, where we first agree on the list of newcomers reflected in the metrics and events organized, and then we send the survey to those newcomers. Qgil-WMF (talk) 07:02, 11 October 2017 (UTC)

Point noted here (https://phabricator.wikimedia.org/T177522) under Survey section. User:SSethi_(WMF) 00:19, 12 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

"Make more documentation"

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

I wonder whether we should keep that recommendation, considering that it si so generic that it is not actionable. Qgil-WMF (talk) 15:25, 10 October 2017 (UTC)

I feel like it is a known fact and there is a recommendation already about looking into improving documentation on a specific topic in the key findings.  User:SSethi_(WMF) 20:07, 10 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Merging "Recommendations for next steps" into "Key findings"?

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

These two separate sections seem to have the same goal. Why not joining them? Some items under "Recommendations for next steps" could be shorter. Qgil-WMF (talk) 15:29, 10 October 2017 (UTC)

Made this change accordingly. User:SSethi_(WMF) 20:01, 10 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

55.65% when n=9

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@SSethi (WMF): "55.65% of respondents of this survey" - did you mean 55.55%? AKlapper (WMF) (talk) 16:28, 10 October 2017 (UTC)

We removed it anyway, so this is resolved now :) User:SSethi_(WMF) 18:16, 10 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

"devoid"

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@SSethi (WMF): "Fix the IP ban that devoid access to Phabricator" - Looking up "devoid" I wonder if that is the right term, maybe "blocks"? Also might welcome a comment that this refers to specific providers in specific areas, to provide context for the average reader, and potentially linking to https://phabricator.wikimedia.org/T167915#3565329 ? AKlapper (WMF) (talk) 16:31, 10 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Full survey analysis page

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

IMO there is no value in this page User:SSethi (WMF)/New Developers Quarterly Report (July - September 2017)/Survey analysis, do you see any? It is not conveying anything extra other than the graphs which can be anyways hard and dangerous to interpret with n=9. Should we get rid of this page? User:SSethi_(WMF) 20:44, 10 October 2017 (UTC)

I would keep it as documentation, of course. Qgil-WMF (talk) 07:00, 11 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Good to go!

The following discussion is closed. Please do not modify it. Subsequent comments should be made on the appropriate discussion page. No further edits should be made to this discussion.

---

@SSethi (WMF), @AKlapper (WMF) I think the New Developers/Quarterly/2017-10 report is good to go! Qgil-WMF (talk) 06:59, 11 October 2017 (UTC)

Yay! we are live :) User:SSethi_(WMF) 21:46, 11 October 2017 (UTC)

The discussion above is closed. Please do not modify it. No further edits should be made to this discussion.

Possible action items

• Review the tutorials explaining the code contribution process and investigate what we can improve further. Such as: include a list of easy tasks requiring one-line fixes, etc. that might bring some value to the first contribution made by newcomers
• Getting Wikipedia Android Apps project listed as a featured project on the New Developers page
• Better tracking information of new developers
  • Quim's idea to start with developers from workshops in Africa via CRM
• Shorten the survey, change the timeframe, and send it not just at the end of the quarter but a few times?
• Can we do anything about this issue "Fix the IP ban that blocks access to Phabricator"?

@AKlapper (WMF) @Rfarrand (WMF) @Qgil-WMF What do you think? User:SSethi_(WMF) 15:42, 12 October 2017 (UTC)

@SSethi (WMF):

• Having real one-liner tasks (e.g. typo fixes) offered instead of just "check out Examples and make a change that does not fix any real bug" is exactly what https://phabricator.wikimedia.org/T146960 is about. We'd need folks to do this (categorize #easy tasks, I can try to remember but it's hard to judge if you're not the dev), and then using that in https://phabricator.wikimedia.org/T161901.
• Survey timeframe: Up to you, but if you want to go for monthly sending out, please update our docs in https://phabricator.wikimedia.org/T177522
• Re Phab IP ban: It is meant to be only about specific Wikipedia Zero providers. Which might have bugs which someone needs to fix. Such as https://phabricator.wikimedia.org/T173537 or https://phabricator.wikimedia.org/T174342. To read more about the IP block, see https://phabricator.wikimedia.org/T170200#3424774... AKlapper (WMF) (talk) 16:20, 12 October 2017 (UTC)

@SSethi (WMF) Thanks so much for putting this together. It't really valuable. :)

• For point #9: perhaps if this is the case, we can hold yearly (??) "tell your friends about contributing to Wikimedia" volunteer recruting drive. Maybe using similar tactics to how other non-profit organizations ask for money.
  • We could write both an email and a facebook post with links that people could copy and paste and then just send that around
  • We could ask people to think about people that they know that might like to get interested & if they don't want to do anything - we could approach them ourselves. "Hi NAME, we heard from Quim Gil that you might be interested in learning about volunteer opportunities to contribute to............... if you have any questions about getting started, please don't hesitate to ask"
  • We could even be crazy and ask Fundraising to add some details in some of their banners "Donate money OR volunteer your developer skills"
• For point #8: maybe we can send personal messages to some newcomers asking them to fill it out next time. Shortening it, like you plan, is probably going to help also.
• For point #7: certainly we should, as a team, look at what questions we want to ask people during hackathon registration. Registration for Barcelona will sneak up on us pretty quickly here.
• For point #3 can we work with the Community Tech team to set up a "newcomers-code-review-day" once every quarter? Or even make it wikitech-l wide and send an email out with code written by newcomers that has not been reviewed and / or needs help?
• I think we should have an action item to check in with other small hackathon organizers (Ghana, Greece, Netherlands, etc) both just a week or so in advance of their events and again afterwards.
  • The pre event check in would be to offer any last minute support in any capacity
  • The post event check in should be to see if they want to write a blog or if they found our tasks useful, or if they discovered other tasks that still need work, or if there is anything else they want to tell us. We should also ask them where they are collecting their lessons learned and add a link to that to our hackathons documentation. Maybe also, specifically, what they wished they had known in advance of organizing the event. Rfarrand (WMF) (talk) 23:37, 13 October 2017 (UTC)

@AKlapper (WMF) @Qgil-WMF & @SSethi (WMF), any thoughts? ^^ :) Rfarrand (WMF) (talk) 23:40, 13 October 2017 (UTC)

I'm wondering.. if we should make the following tasks as subtasks of https://phabricator.wikimedia.org/T178630

• Gerrit/Tutorial https://phabricator.wikimedia.org/T161901
• Phabricator IP/Ban https://phabricator.wikimedia.org/T173537
• CRM for TC: Start with new developers from workshops in Africa https://phabricator.wikimedia.org/T177583
• Follow up with organizers of WikiFemHack?
• I created already one about making Wikipedia Android app as a featured project for new developers https://phabricator.wikimedia.org/T178633
• Shortening the survey is already now a TODO in https://phabricator.wikimedia.org/T178627

@Rfarrand (WMF) All your points 1-3 are some promising ideas for improving outreach, that we could utilize in https://phabricator.wikimedia.org/T176488

@AKlapper (WMF) Better triaging for easy tasks sounds interesting, but I am not sure if we currently have the bandwidth to work on this, and if we should carve out a tiny bit of it for the purpose of improving the Gerrit tutorial. User:SSethi_(WMF) 22:44, 19 October 2017 (UTC)

@SSethi (WMF): Re Follow up with organizers of WikiFemHack: https://phabricator.wikimedia.org/T176861 might be usable for this (as that task needs an update anyway) AKlapper (WMF) (talk) 14:23, 25 October 2017 (UTC)

Slight change > Make new tasks as subtasks of https://phabricator.wikimedia.org/T167085 as Quim suggested in T178630#3699139 User:SSethi_(WMF) 20:20, 23 October 2017 (UTC)

Looking at https://phabricator.wikimedia.org/T167085 which is called "Release a "New Developers Quarterly Report" including metrics, trends, lessons learned, and recommendations by Oct 2017", those tasks listed above do not block releasing a Quarterly Report by Oct 2017 and that task is already resolved. Hence they are not subtasks (tasks that need to be done to resolve a parent task) in my understanding - also see Phabricator/Help#Parent tasks and subtasks. So I guess I disagree with Quim here. :) AKlapper (WMF) (talk) 13:51, 25 October 2017 (UTC)

Outreach events retention

Hello,

Thanks for compiling this report − very interesting!

I am curious about the retention of events. How is the number of developers attracted (eg for the Wikimedia Hackathon, 37) computed? Is that based on the registration survey? (For comparison, the report of that event mentions 56 newcomers)

From there, the 3 retained is based on a Kibana query.

• the query only contains 19 usernames. I guess it is because we don’t have the usernames of the 18 others?
• the query appears to look through Gerrit data ; is that a deliberate choice? I would have thought that contribution to any Technical space would count as retained − the same query against Maniphest appears to return 5 users. Jean-Fred (talk) 07:08, 19 October 2017 (UTC)

@Jean-Frédéric Thanks for your question! Yes, we are relying on the registration survey. Members of TC went through the events registration spreadsheet to remove folks who may not be new developers such as staff members, who we didn't see at the event, folks who have been already active for a while, etc. The final list had 37 developers (from Vienna). We tried to look up for more information for people (whom we didn't have any Phabricator, Gerrit username info) in Wikimedia Biterg using their email address. Then we could gather usernames of only 19 people. (some more info here)

For this report, we have target new developers who have contributed at least one patch in the last 90 days. For the survey that we sent out, for the metrics we gathered, we considered the same audience. And, so the query containing 19 usernames looks through Gerrit data. 

Having said that, yes we need to improve on measuring this data better, key finding 7 in the report New Developers/Quarterly/2017-10#Key findings :)

@AKlapper (WMF) might be able to shed more light on this if I've missed anything as he is the master mind behind pulling all this together :P User:SSethi_(WMF) 21:00, 19 October 2017 (UTC)

@Jean-Frédéric: Good questions. And thanks for your interest. :)

In addition to what Srishti already wrote:

19 usernames indeed because we could not find any activity in the data for the 18 other usernames. But that might be because https://wikimedia.biterg.io for example does not cover activity on GitHub where some projects are located.

Concentrating on Gerrit data became a choice at some point. I first queried on the mainpage of https://wikimedia.biterg.io for the usernames but then had to realize https://phabricator.wikimedia.org/T177566#3663047. The pleasure of working with systems that you need to understand. :P Also note that the Maniphest query will only show you users who created a task, but not users who for example commented on a task. See for example https://phabricator.wikimedia.org/T161926 or https://phabricator.wikimedia.org/T161928.

And https://phabricator.wikimedia.org/T175854 provides even more technical details about the Kibana queries that you might not want to dive into. :) AKlapper (WMF) (talk) 17:11, 24 October 2017 (UTC)