Wikimedia Release Engineering Team/20180320 Selenium Retrospective

Retrospective: Selenium Ruby to NodeJS project

Attendees: Jon Robson, Željko Filipin, Jean-Rene Branaa, Erik Bernhardson, Elena Tonkovidova, Timo Tijhof, Jan Drewniak, Joaquin Oltra Hernandez

Agenda

Brief overview of project
- Purpose/Objectives

Zeljko: Were using old Ruby framework from the onset. Worked for some, not for others. Framework was perhaps overly complicated. Additional abstraction with Cucumber turned out not to be needed by most.

Roundtable

What went well

Timo: Worked from master from the start and happened on every commit. Works locally and doesn't make assumptions.
Jon: some repos were doing this: MobileFrontend and Minerva
Erik: Also because phantomJS is a giant pain.
Erik: cirrus was also doing per-commit, although external from Jenkins.
Timo: industry was moving forward faster than we could.
Jan: Headless browser support also positive.
Zeljko: SauceLabs was a nice feature for Ruby, providing Windows/IE support, but in the end was also causing daily failures, and the Windows support wasn't as important (most issues were found in Firefox and/or Chrome as well).
Jon: Communication was pretty good during the switchover. It was clear what was happening and when.
Elena: +1 to Jon. Zeljko is super helpful with any questions and all explanations of what and how works

Timo: Feels like the new framework is easier to use. The new framework allow you to use both BDD and TDD. It feels more naturual to JS developers.
Elena: Any experience with Rspec?
Timo: I've not, but it is very similar to Mocha.
Zeljko: Stuck with simple builtin assertion library to keep things simple.
Erik: we ended up using chai.js assertions library, probably similar to mocha.
Timo: QUnit library
Erik: Not sure why we used Chai vs Mocha, but it may have been the error messages.
Elena: Chai assertions is more sophisticated.

What could have gone better

Jon: Strategically it made totally sense to switch from Ruby to Node, but our team has failed to port them over and is stuck to see an easy way to do so. I wrote up my experiences here > https://www.mediawiki.org/wiki/User:Jdlrobson/Experiences_porting_Ruby_browser_tests_to_NodeJs
Jon: Team has felt an additional stress now that Ruby is not supported and we're dependent on it. 13+ tests in MobileFrontend, 75+ in Minerva.
Jon: We've stopped writing tests :/ Jon: Webdriver has a bit of a learning curve even for those who know JS (sync nature is not obvious)
Erik: There was a lot of effort involved in the porting of tests. Hopefully we'll see some benefit over time.
Zeljko: Ruby as a language was perhaps a barrier to entry (learning new language, in addition to learning Selenium and the PageObject pattern)
Zeljko: Seemed like some teams waited until last warning to migrate.
Elena: When migrating the tests did you (Erik) review the tests themselves or just migrate them?
Erik: just migrate. Pretty confident in our tests as they stood.
Jon: Not everyone had a plan for the port. As a result new test development stopped. Not sure if Ruby and NodeJS could coexist. Extensions and libraries that they depend on in Ruby aren't necessarily available on NodeJS.
Zeljko: There's no technical reason that he can think of that you couldn't port tests one at a time. So they should be able to coexist.
Jon: Also motivation for migrating is low as the Ruby tests are working.
Zeljko: willing to pair and help migrate. However, wants to have each team responsible for their own migration in order to insure teams are comfortable with maintaining them moving forward.
Erik: If using Cucumber, they were able to migrate one feature file at a time.
Jon: Didn't know that was an option (Cucumber) as he thought we are abandoning Cucumber.
Zeljko: that was the plan. Level of abstraction didn't seem worth it for developers.
Elena: Even with an understanding of the code, Cucumber is still valuable for communication/overview.
Zeljko: Are there things that are missing in the NodeJS framework that I've not already captured?
Jon: Multi-user logins (as user a do x as user b do y).
Zeljko: Dan did a lot of work with the helper files, it wasn't clear how much was there and who was using what. As a result not everything was ported. It was developed over 5 years.
Jon: It might be too much to ask to complete the transition over a few months (maybe need to let the NodeJS evolve over time too).
Jan: Adding more abstraction capabilities as Jon mentioned.

Tips

Erik: Built-in assert() wasn't providing useful error messages. Chai provided these. Mocha would work as well.
Joaquin: https://github.com/power-assert-js/power-assert
Timo: Built-in assert() is indeed quite basic. It does have additional methods with better messages though - https://nodejs.org/api/assert.html , but nothing as sophisticated as Mocha or Chai indeed. There's also https://www.npmjs.com/package/assert-diff

Actionables

Zeljko: Current activities (selenium improvements) - https://phabricator.wikimedia.org/T182421
Zeljko: needs help with code reviews and such to wrap up project in next couple of days.
Timo: Willing to help with code reviews.
Joaquin: Sharing experiences on a regular basis would be helpful.
Erik: Willing to show/review how cirrus integrated cucumber/node
JR/Zeljko: Summary, actionables, writeup by EOW
JR: Reduce future retrospectives to 60 min.
JR/Zeljko: next update/sync-up meeting 50min, then down to 20min.

Chat Transcription

Jon Robson
8:09 AM
we did but only for some repos

Erik Bernhardson 
8:10 AM 
also because phantomjs is a giant pain

Jon Robson
8:10 AM
we are using Ruby on MobileFrontend and Minerva per commit

Erik Bernhardson
8:11 AM
cirrus was similarly doing ruby per-commit, although external from jenkins

Jon Robson
8:17 AM
What is the format of this meeting Jean-Rene ? Sounds like we're going over things that went well?

Jean-Rene Branaa
8:17 AM
yup

Timo Tijhof
8:17 AM
https://etherpad.wikimedia.org/p/20180320-SeleniumRetrospective

Jon Robson
8:21 AM
i can expand on my points
yeh sorry for dumping in etherpad just not sure how and when to jump in

Elena Tonkovidova
8:24 AM
Jan: makes sense

Jon Robson
8:29 AM
i guess also the motivation is low for us - the Ruby tests are working and finding bugs
1) I think setting up those repos with Node and Ruby browser tests in parallel would be a good starting point to incrementally porting tests
2) I'd love to get some insight into experiences using cucumber and just porting the step definitions. Particularly around how you got round the Selenium wikimedia library (e.g. log_in is a method in Ruby - I think that's a lot trickier in Node.js?)

Elena Tonkovidova  
8:36 AM  
we should have Tips on etherpad, so the above can be added

Jean-Rene Branaa
8:36 AM
I'll capture the chat at the end and put it in the meeting notes

Jon Robson
8:36 AM
Thanks JR!

Jean-Rene Branaa
8:36 AM
NP

Elena Tonkovidova 
8:38 AM 
Jon: re: 2) - it depends on how Cucumber files are actually structured. Sometimes they are so elaborate that porting them to not-so-familiar framework is quite cumbersome

Erik Bernhardson
8:42 AM
worth noting cirrus tests are more "integration" than browser, primarily testing api and end-to-end integration

Elena Tonkovidova
8:45 AM
Erik:exactly. I looked at your tests - unfortunately, historically our tests are elaborate and involved user scenarios
Erik Bernhardson

8:48 AM
we ended up using chai.js assertions library, probabily similar to mocha

Elena Tonkovidova
8:51 AM
chai assertion is more sophisticated than mocha assertions

Joaquin Oltra Hernandez
8:53 AM
this could be useful https://github.com/power-assert-js/power-assert improves node's assert messages

Željko Filipin
8:53 AM
https://phabricator.wikimedia.org/T182421

Joaquin Oltra Hernandez 
9:02 AM 
testing support group

Jean-Rene Branaa
9:02 AM
yup

Elena Tonkovidova
9:02 AM
sounds great!
SIG

Jon Robson
9:07 AM
Write up experiences!

Joaquin Oltra Hernandez
9:07 AM
quarterly sounds good

Elena Tonkovidova
9:07 AM
Successes and failures

Joaquin Oltra Hernandez
9:08 AM
etherpad died
it's like we never met

Jean-Rene Branaa
9:09 AM
lol

Joaquin Oltra Hernandez
9:16 AM
1h makes more sense IMO

Timo Tijhof
9:16 AM
the etherpad has returned