Topic on Talk:Wikimedia Technical Committee/MediaWiki Platform Architecture Principles/Draft 9-2018

Software development SHOULD follow the Continuous Integration model

2
Tim Starling (WMF) (talkcontribs)

This stands out as one of the few principles that has negative cost, since CI is effectively a declaration that we're not going to bother doing manual testing of our software before deployment. It's based on the myth that all bugs can be found automatically. Tests are run after every commit, which means that they have to run quickly, which leads to the ironic situation of having a "continuous integration" system in which very little integration testing is actually done. Instead we run unit tests, in which units are tested against the assumptions of the developers who wrote them. External dependencies are mocked out, replaced by stubs which work the way the developer thinks they should work, instead of the way they actually work. As a result, severe bugs are often noticed by users after deployment.

However, I'm all for cheap things. The reason CI works for us is because in our particular environment, mistakes can be quickly fixed, and nobody dies if we get it wrong. Both users and developers appreciate the fast deployment of features and bug fixes. I'm just unsure about elevating it to the status of an architectural principle. It seems to me that it is a development methodology, not an architectural principle. It's a methodology that is fairly specific to software that we develop and deploy in-house. Nobody is suggesting that we continuously integrate upstream changes from Debian Unstable — it is only feasible when development and deployment are closely tied together. It's fine as a statement of fact for MediaWiki, but it seems too broad to apply to all "other Wikimedia engineering endeavours".

DKinzler (WMF) (talkcontribs)

I agree - having unit tests is a requirement in my mind, but they are not a replacement for integration tests or system tests. We need to have all of these.

"good test coverage" is an engineering practice, but not really an architecture principle. design for testability is., but we already have that on another line.

In the old version of the document, we had a section called "Engineering Process Considerations". I'll revive that and move the points about testing there.

Reply to "Software development SHOULD follow the Continuous Integration model"