Core Platform Team/Development Practices/Testing

Background

During the CPT remote offsite in December 2019, we had a session about our current state of testing. In this session, the team had a chance to discuss some of the challenges people encountered when writing tests and the various approaches people took when writing new tests.

We learned that people were less likely to write tests if:

• Tests required too many mocking of other classes
• There was no accountability
• Testing was not made a priority
• People felt like they didn’t have the time to write tests
• People were unsure where to start
• The tooling made it harder to test for certain cases

As we began to think of ways to help the testing culture we realized the problem was two-fold. We had to create a process for all new features to make sure we weren’t accruing technical debt and a strategy to tackle the testing debt. Both strategies can be found below.

Process For New Features

Starting January 2020, all new features built by our product team(s) should have unit and integration tests or a linked Phabricator ticket with the TESTING-ROADBLOCKS tag explaining why testing wasn’t possible. We understand that there will be times where testing is difficult or isn’t possible (yet) and we want to be able to record those instances to share back with the Release Engineering Team.

How will this work

Product managers and tech leads that are writing Phabricator tickets for new features will be required to include unit and integration tests as requirements for closing a ticket. As engineers pick up tickets we ask them to first check if unit and integration tests are listed as a requirement if not please do add them. We also ask engineers to NOT merge patches without unit and integration tests or a ticket stating why testing wasn’t possible.

Check-Ins/Retros

We will check in with the product team(s) once a month for the next quarter to get feedback on the process and adjust accordingly.

Testing Debt

After discussions with JR from RelEng, we realized this problem is larger than we anticipated. So we want to spend the next quarter gathering more information about the challenges people encounter and have more discussions with others that have tried to lessen the testing debt. We plan to use the tickets from TESTING-ROADBLOCKS to aid in those discussions. We also plan to join the RelEng testing working group.