Developer Satisfaction Survey/2024/Code review
📖 Developer Satisfaction 2024 Report
💯 Code review
edittl;dr
- Time spent on code review – The majority of volunteers spend <10% of the time doing volunteer Wikimedia development reviewing others’ code, while staff most commonly spend 10-20% of their work week reviewing others' code.
- Code review quality – Similar to last year, respondents say feedback is more helpful than timely.
- Gitlab – Only 46% say they are satisfied with Wikimedia's Gitlab, lower than the 66% last year who said they planned to continue using it.
- Gerrit – 64% say they are satisfied with Wikimedia's Gerrit, similar to the 60% last year who said they planned to continue using it.
We asked survey takers if, in the past year, they have had their code reviewed by someone else and/or reviewed someone else's code.
68.9% of respondents either reviewed code, had their code reviewed, or both.
# Time spent on code review per week
We asked non-staff survey takers, “In the past year, on average, what percentage of your time doing volunteer Wikimedia development work was spent reviewing others’ code?”
The majority of respondents (69.8%) said they spend less than 10% of their volunteer Wikimedia development work time each week reviewing others' code.
We asked staff survey takers, “In the past year, on average, about how many hours per week did you spend reviewing others’ code as part of your role as a member of the Wikimedia developer community?"
There was a wide distribution of responses, but the majority of respondents (56%) said they spend at least 4 hours per week reviewing others’ code. The most common response was 4-8 hours per week, or about 10-20% of their work week.
# Code review quality
We asked survey takers how often the feedback on their code is helpful and/or provided in a reasonable amount of time (as a member of the Wikimedia Developer Community during the past year).
- The majority (89%) of respondents indicated that feedback is always or often helpful
- Less than than half (49%) said that feedback is always or often timely
# Feedback on the code review process
We asked survey takers, “Please share any other feedback you may have about the code review process”. Some themes emerged from the answers.
It‘s slow
5 out of 42 answers (12%) thought the process was slow.
Code review process is too slow. Patches shouldn't take years to get merged. New contributors generally won't come back to address comments offered after months.
# Gitlab satisfaction
We asked survey takers “Thinking of your experience in the past year, how satisfied are you with Wikimedia’s Gitlab?”
- 46% said they were satisfied with Wikimedia’s Gitlab
- 25% said they were neither satisfied nor dissatisfied
- 27% said they were dissatisfied
- 2% were unsure
# Gerrit satisfaction
We asked survey takers “Thinking of your experience in the past year, how satisfied are you with Wikimedia’s Gerrit?”
- 64% said they were satisfied with Gerrit
- 11% said they were neither satisfied nor dissatisfied
- 25% said they were dissatisfied with Gerrit
# Gerrit vs GitLab satisfaction by role
This chart compares satisfaction with Gerrit by functional role vs satisfaction with GitLab by functional role sorted by satisfaction with Gerrit.
# Gerrit activity
We asked respondents who said they use Gerrit, “In the past year, about how many commits have you made in Gerrit?”
Over one-third of respondents (38.4%) said they had made more than 100 commits in Gerrit last year. About a quarter of respondents (26.7%) said they had made 11-10, and another quarter (26.7%) said they had made 1-10.
We looked at (self-reported) Gerrit activity level by tenure. (Self-reported) commit activity was higher for respondents with longer tenure. Almost half (48.9%) of respondents with 10+ years tenure said they had made more than 100 commits in Gerrit last year.
# Gerrit satisfaction (by Gerrit activity level)
We looked at Gerrit satisfaction by respondents’ Gerrit activity level (i.e., by respondents’ self-reported number of commits made in Gerrit last year).
Respondents with higher levels of activity in Gerrit were much more satisfied with Gerrit. 70% of respondents with 100+ Gerrit commits last year were very satisfied with Gerrit, compared to just 7% of respondents with 1-10 Gerrit commits.
Weighting by Gerrit activity
editFolks who took this survey are more active users of Gerrit than most.
We know this because we know how many commits every user of Gerrit has made over the past year, and we can compare that to the self-reported Gerrit activity from this survey. When we do that, we find folks who took this survey are more active than most Gerrit users.
.png)
Using their self-reported Gerrit commits, we can weight answers to better-represent the population of Gerrit users (by Gerrit activity). So, responses from folks with a lot of Gerrit commits end up counting a little less, and responses from folks with fewer Gerrit commits end up counting a bit more.
We cannot say for certain whether 1) more-active Gerrit users are more likely to answer the survey (due e.g., to a higher likelihood of seeing the survey advertised on Gerrit), as we suspect is likely, or whether 2) survey respondents systematically overstated their Gerrit activity. However, we think these weighted estimates are useful context for understanding how responses might change if we were able to survey every Gerrit user.
Weighted satisfaction
When we apply the weighting (i.e. when we downweight the responses of highly active Gerrit users, and upweight the responses of less active Gerrit users), overall Gerrit satisfaction decreases from 64% to 57% satisfied; and overall Gerrit dissatisfaction increases from 25% to 32% dissatisfied.
.png)
When we look at Gerrit satisfaction broken by function role group and staff/non-staff, with the weighting applied, the satisfaction of each functional role group stays basically the same, as does the satisfaction of staff and non-staff.
# Feedback on code review tools
We asked survey takers “Please share any other feedback you may have about code review tools”. Some themes emerged from the answers.
GitLab is bad
14 of 58 (24%) open answers lamented GitLab.
Gerrit has a steep learning curve, but it does all the things we need, and it does them well. Getting gitlab to the same level for the core repo seems like a huge effort.
GitLab is an acceptable facsimile of GitHub, but GitHub isn't very usable, especially for this workflow. I've migrated one of my tools there, but I can't imagine using it for a project the size of MediaWiki.
Gerrit should stay. Gitlab can't fully replace it. It's a myth that volunteers left because of Gerrit. The real reason they left is not getting reviews, which is not solved by replacing the review tool but is a social issue.
Gerrit is bad
7 of 58 (12%) of open answers bemoaned Gerrit.
Even after having used Gerrit for several years I still find a lot of things unintuitive.
I have not made that many commits myself in gerrit, but I work with teams who use it. [...] Gitlab is wonderful and we can't move toward using it fast enough!!