Growth/Engineering/Incident tracking

This page tracks "incidents", meaning notable, user-facing bugs, for code in GrowthExperiments. We are tracking these to assess frequency of issues and decide what/how we need to adjust practices in response, not to cast judgment or blame.

Criteria for adding tasks:

  • would have blocked the train, but fixed in advance
  • blocked the train
  • incorrect/missing analytics for Growth features
  • significant performance change

Criteria for low/medium/high:

  • high: blocked train, caused outage in production
  • medium: ?
  • low: ?
Date Task Tagged with #wikimedia-production-error Train rollback How the issue was found Lessons / Follow-up task(s) Impact Severity
2022.06.23–2022.07.04 T311916: "Add an image" structured edits add a blank line instead of an image no no Patroller on cswiki informed Martin, Martin informed the team. end-to-end tests would have prevented the problematic patch in VisualEditor from being merged in the first place. all edits saved with the AddImage plugin contained a blank line, rather than an image and caption. high
2022.06.23- 2022.06.29 T311467: [wmf.17-mobile] "Suggestions" label has no padding no no Manual QA by Elena Visual regression or e2e testing could catch this. No close button for mobile users of structured tasks, semi-broken UI medium
2022.03.10 – 2022.05.25 T309152: [regression-wmf.13] Add image: Missing "View image details" link no no Found by Marshall and Kirsten in manual review of the feature
  • Rely on custom CSS classes rather than upstream provided classes
  • Visual regression or Selenium tests might have caught this issue.
"View image details" button was missing on caption step of "add image" plugin, which makes writing captions more difficult. medium
2022.03.08 T303251 Error: exception of type Error: Cannot unpack array with string keys no no apitesting-test jobs failed for GrowthExperiments. task T303255 perhaps core api-testing job should also run api-testing tests for extensions/skins; that would have caught this in the core patch that caused the breakage. betalabs was broken for some help panel guidance message requests for a few hours. If undetected, this would have been a wmf.26 train blocker. low
2022.02.24 T302746 Post-edit dialog does not appear for newcomer task edits no no Rita and Marshall noticed on testwiki, and commented in Slack on February 24. We currently have no end-to-end tests verifying the presence of the post-edit dialog. (A patch pending review will this test.) That wouldn't have caught the problem in VisualEditor's patch approval, but would have when GrowthExperiments patches would be checked. Newcomer experience degraded, miscellaneous bugs due to post-edit dialog not loading, medium
2022.02.10 T301486 Add an Image: back/next buttons in onboarding screens are not instrumented no no Morten, while doing analysis on add image. missed during code review and QA, not a regression. Clicks on back/next buttons in onboarding dialog were not instrumented, so we don't have data to help better understand how users interact with this dialog. low
2022.02.06 T301062 Error: Call to a member function getId() on null no no @Sunny00217 on zh betawiki side-effect of the MentorProvider refactoring, where an important check was unintentionally removed. Follow-up patch adds a regression test. zh betawiki was broken for new user accounts. This would have been a train blocker, similar to the December 15 2021 incident. low
2022.01.27 T300336 Add an image: Mobile-only quality gate shown to desktop users no no Marshall and Elena, after the feature flag was enabled in production. A small subset of newcomers who are opted into imagerecommendation variant would have experienced a confusing dialog when attempting to do image suggestion tasks on desktop. medium
2021.12.15 T297827 Error: Call to a member function getId() on null yes yes Antoine, after train promotion to group1 Newcomers were unable to login on cawiki for about 25 minutes. group1 wikis were rolled back to previous train version. high