Developer Satisfaction Survey/2024/Continuous integration
📖 Developer Satisfaction 2024 Report
🔁 Continuous Integration
edittl;dr
- Queue time satisfaction – Only half of respondents are satisfied with CI queue time, similar to last year.
- Overall satisfaction – 64% are satisfied with CI overall, slightly higher than the 57% last year who said were satisfied with CI features aside from queue times.
# CI use in the past year
We asked survey takers, “In the past year, have you used Wikimedia’s continuous integration?”
- The majority (52.4%) of respondents said they used Wikimedia's CI in the past year.
- 42.2% said they did not use it in the past year
- 5.4% were unsure
# CI queue time satisfaction
We asked survey takers, “How satisfied are you with the Continuous Integration queue time (how long it takes to report test results in Gerrit)?”
Of the survey takers who indicated that they did use Wikimedia's CI in the past year,
- Half (50%) said they were satisfied with CI queue time
- 14% said they were neither satisfied nor dissatisfied
- 30% said they were dissatisfied
- 6% were unsure
# CI overall satisfaction
We asked survey takers, “How satisfied are you overall with continuous integration?”
Of the survey takers who indicated that they did use Wikimedia's CI in the past year,
- a majority (64%) said they were satisfied with CI overall
- 14% said they were neither satisfied nor dissatisfied
- 16% said they were dissatisfied
- 6% were unsure
# CI onboarding satisfaction
We asked survey takers, “How satisfied are you with onboarding new people into continuous integration?”
Of the survey takers who indicated that they did use Wikimedia's CI in the past year, the most common response was uncertainty: 39% were unsure if they were satisfied with CI onboarding.
- only (19%) said they were satisfied with CI onboarding
- 24% said they were neither satisfied nor dissatisfied
- 28% said they were dissatisfied
# Feedback
We asked survey takers, “Please share any other feedback you may have about Wikimedia’s continuous integration” Some themes emerged from the answers.
Finding test errors
5 out of 53 responses talked about difficulty finding the cause of test failures in the test output.
it can be hard to find the real error that caused the CI run to fail in hundreds of lines of logs
It would be nice if the actual failure was noted on Gerrit
Confusion
8 out of 53 open answers found some aspect of our continuous integration confusing and/or under-documented.
We shouldn't have to tell people to type magic words like 'recheck' or 'check experimental'!
It's very unclear which tests run on which repo, and why.
[new developers] have difficulty understanding why they received errors, or what to do about it.