Developer Satisfaction Survey/2024/Shared test environments
📖 Developer Satisfaction 2024 Report
🧪 Shared test environments
edittl;dr
- Use – A majority of respondents used a shared testing environment
- Beta – 45% are satisfied with beta, up from 38% in 2023
- Pre-merge – 50% are satisfied with their ability to test in a shared environment pre-merge
- Priorities – respondents ranked their shared testing environment priorities
Shared test environment use
We asked survey takers, “In the past year, have you used any shared test environments (e.g., beta cluster or patch demo)?” A majority of respondents (54.5%) said they had not used a shared test environment in the past year.
Shared test environment use (by role)
Shared test environment use (by tenure)
# Beta satisfaction
We asked survey takers, “If you have used the Wikimedia beta cluster, how satisfied are you with the beta cluster?”
Of those who said they had used the Wikimedia beta cluster:
- 45% said they were satisfied
- 18% said they were neither satisfied nor dissatisfied
- 32% said they were dissatisfied
- 5% said they were unsure
# Beta satisfaction (by role)
Shared test environment satisfaction
We asked survey takers, “How satisfied are you with your ability to test your code in a shared environment before merge?”
- 50% said they were satisfied
- 11% said they were neither satisfied nor dissatisfied
- 34% said they were dissatisfied
- 5% said they were unsure
Shared test environment use cases ranked by priority
We asked survey takers, “How would you rank the following use cases for a shared test environment? Rank the following from 1 (most important) to 5 (least important).”
Respondents were asked to order each of the following use cases by priority:
- Exploratory testing/QA of recent patches
- Demoing/showcasing
- Test automation of end-to-end tests
- Test automation of integration tests
- Test automation of unit tests
- Other
These were presented in a random order to each respondent.
The chart below shows descending order of average rank. Like last year, "Exploratory testing" has the highest average rank and received the most #1 rankings. The use cases with the average lowest rankings were "test automation of integration test" and "test automation of unit tests". The graph does not capture "other".
# Feedback
We asked survey takers, “Please share any other feedback you may have about shared test environments” Some themes emerged from the answers.
Folks like PatchDemo
PatchDemo is a great tool, and it would be great if it could be improved to work for more code bases and use cases.
Beta
Beta is useful for finding terribly bugs that would knock over production, but that's about it. Most of the "interesting" real world test cases for our code involve user content, and that's not present on beta and painful and difficult to import.