Wikimedia Discovery/Meetings/Portal retrospective 2016-04-22

Discovery Portal Retrospective

2016-04-22

Covering whatever has happened related to the team since the last retro (2016-03-23)

• Quarter ended; new quarter started
  • Process for deciding on new quarterly goals
• Full month without Julien
• Deb and Jan traveling
• Fixed a bunch of bugs
• Ran another test
• Brainstorm/roadmap meeting
• Moiz begins his transition out
• Weeklong code freeze due to codfw migration
• Started using new portal backlog workboard

Review action items from before

• Jan: Document "production quality", beta site, demo sites, etc.
  • Generally in progress, but work related to this task:
  • Portal A/B Testing wiki has been updated with a link to the 'archive version' of the A/B test, as well as a link to the analytics results. https://www.mediawiki.org/wiki/Wikipedia.org_Portal_A/B_testing
  • Created a url to show 'in-progress' work to team: https://people.wikimedia.org/~jdrewniak/in-progress/index.html
• Kevin: Try to unstick quick surveys
  •  Done, although I'm not sure if they are re-stuck or just on hold. Either way, they are no longer the responsibility of the portal team.
  • Agreed - the work done by the Portal team is done for the time being. All survey tickets have been moved to the Portal Backlog board

What went well?

• Ran the localized top-links A/B test
  • Some delays turning it on and turning it off
  • Showed actual user impact
  • Went smoother than earlier tests
• Working with Julien on code review is going well
• Jan has done very well with the tasks assigned and delving into upcoming tasks and research +1+1
• Great analysis by Mikhail - he's been busy with lots of other tests and work +1+1
• Good conversations (that Deb had) with Legal, Comms, and User Testing teams about upcoming work +1
  • Also, Edward was very helpful with qualtrics
• Portal work (small bug fixes) in Tech News, weekly status updates, mailing lists. (good communication) +1+1
  • Reminder: Work that seems small to you might still be worth publicizing
• Good collaboration with User Research team
• Portal Brainstorming meeting was good - lots of new ideas that can be acted on in the future+1+1
• Communication between Portal engineers and analysis team+1
• Team has gelled and supported each other well through some organizational turbulence

What could we improve?

• A/B Test roll out process
  • We have improved, but could still be improved more
  • Checklist for all the things needed to complete a test
    • We do have a checklist which is very valuable; always room for improvement
    • Testing checklisst: https://meta.wikimedia.org/wiki/Discovery/Testing
  • Look at how to avoid past problems in the future (retro for each test?)
    • Would be cool to know how long each step took, so we would know where to optimize
  • Maybe sit down with FR team, because they run tests very quickly (e.g. 30 minutes)
    • They are less busy right now
  • Maybe talk with search team about their decision to build relevance forge, as a way of putting things in front of users to get feedback
• The time it takes us to run an A/B Test
  • If all our work is going to be tested, we need to get really good at doing it
  • Some delays were due to developers being new; some was rework of hasty prototype quality code
  • Last test didn't run into those problems, but still seemed to take a while. Not sure why.
  • Portal doesn't have a rich framework like mediawiki does, so have to invent more
  • Reviews (code, security) does take a while. Seems to be getting shorter.
  • Need to be careful because any change affects so many people
  • Perhaps we could get people to review mocks in some cases rather than running a full test
• Survey should've been out by now, disconnect between user research/design and product
  • UR/D came up with a survey they felt good about; when Deb saw it, it didn't align with her expectations
  • We should include product manager in meetings of what we're trying to do with a survey (earlier in the process)
  • The email thread has helped; we will probably do multiple surveys
  • Now facing some unexpected technical issues to implement what UR wants to do
• Future portal meeting was underwhelming (i can provide more context)
  • Was hoping for more excitement about big ideas and ongoing work
  • We need to apply resources where the benefit/cost is highest, so regularly re-evaluating each area
  • Portal team lost months just getting the code into git
  • Team is hitting its stride this quarter, but we still have a lot to learn about what we should be doing
• Design stuff has been slow last two weeks, sorryyy.. New mocks coming today :)
• Moiz shouldn't leave!

What else should be noted?

• Two surveys will probably be run on the Portal in the coming month(s)
  • requires close coordination with User Testing team, Legal and Portal teams
  • two surveys might be done - one to hit up all users after a period of inactivity on the page, the other one to talk directly with users that bounced previously from the portal and engage with a survey.

Action Items

• Jan: Create phab task to wrap up doc of beta/demo/production sites
• Jan: Do a post-mortem on what technical tasks actually took the longest time to complete.

• Deb: create rough mocks of some of the ideas for the future work and add to page on wiki ro further discussion
• Deb: set up meeting with Creative Tech (Megan) to go over their rapid testing process (Deb, Jan, Mikhail)
• Deb: Create ticket for pushing successful lang links / localized phrase to production on portal
• Kevin: Consider some light form of retro after next A/B test