Wikimedia Discovery/Meetings/Analysis retrospective 2016-11-08

Format

edit

Glad/Sad/Mad: http://retrospectivewiki.org/index.php?title=Glad,_Sad,_Mad

Note that "mad" and "sad" don't have to mean literally angry or saddened. They can be used in a playful way as well.

Previous action items

edit
  • Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
    • Include a statement of the perceived problem
    • Still needs to be done
  • Deb: Figure out what to do with the Maps Dashboard
    •   Done - fixed and updated
  • Mikhail: Think about involving GL in sys admin stuff
    • Need to meet w/GL

What has happened since the last retro (2016-09-30)

edit

(using https://www.mediawiki.org/wiki/Discovery/Status_updates as a starting point)

  • Analysis of WDQS geographic breakdown of SPARQL queries
  • Wikipedia.org Portal Dashboard: investigate spike in languages visited
  • Wikipedia.org dashboard: minor updates
  • Bug in current data retrieval script
  • Add comment to search-related eventlogging in iOS and Android apps to inform Discovery of changes
  • Deb has started integrating slowly into the Discovery Interactive Team as a quasi-Product Manager
  • Completed analysis of the results of BM25 AB test (final report)
  • Search results page: how many visitors are on mobile vs desktop -
  • Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
  • Investigate recent spike in pageviews on wikipedia.org portal page
  • Add a PaulScore approximation to discovery.wmflabs.org (waiting on phab:T138087 to be deployed on the train)
  • Compare ZRR for query features across other search engines
  • Investigate what we'd need to do to ignore double quotes in search queries
  • Maps Dashboard: fix and update
  • Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
  • Add a PaulScore approximation to discovery.wmflabs.org

What has made you mad?

edit
  • Interactions with Interactive team
    • e.g. we created a dashboard and have made many improvements on it since its deployment, but the Interactive team does not seem to use it and even cannot remember the URL at times
    • so being asked to learn a bunch of new technologies and do a bunch of additional work to make data available in a different data store so the team can do their own dashboarding is frustrating when current resource isn't being utilized

What has made you sad?

edit
  • Frustration with working with the Interactive team - trying to decide what is needed for metrics
  • Lack of integration between Analysis and Interactive
    • Chelsy and Mikhail are a resource for Interactive to use, and they are being underused

What has made you glad?

edit
  • Chelsy is a super fast learner and has done terrific work on her reports+1
  • Coordination with Search team on test metrics/analysis of tests - quick results
  • Updating the dashboards to be much better
  • M & C learning new things about R and the operations stack to get new and cool technology into our environment
    • Nice to be able to learn during work time
  • Watching this after a tough election night: http://i.imgur.com/lZEeZ8z.gifv
  • Kevin's involvement in Analysis; this team is better for having a TPG person+1+1

What has made you plaid? (aka whatever didn't fit in one of the other categories)

edit
  • There are several IP addresses have been doing weird things to us, e.g. pinging portal million times per day, sending the same search queries thousands of times per day. I know we are generally not blocking any of them, but wondering if we should keep a black list for ourselves, in order to remember to check them when we are doing related analysis so that we can filter out polluted data.
    • Maybe a greylist rather than blacklist, just to identify for reporting?
      • Publicizing the list wouldn't be appropriate (privacy issues)
    • Some cases would be a judgment call; might look useless to us, but maybe they're getting value
  • and...this: http://i.imgur.com/sd15NVu.gifv

Discussion

edit

Interactive team interactions

  • KS: I can see 3 areas: 1) ED-level of what metrics should be tracked, 2) tech level of whether to use grafana or other, 3) inter-team interactions and social component
  • They seem to be making progress on deciding what metrics to track
    • Interactive product folks (Deb/Yuri) should do some early thinking, then bring in analysts to review. Doesn't seem to be going that way so far
  • For tech, there's a ticket to get data into grafana. That's frustrating because we have a dashboard system that we're really happy with, but they won't use it; want us to do additional work and learning to support this other system
    • Our dashboards can do some things that grafana can't
    • Unclear WHY
  • Interactive team feels strongly about doing things their own way. That's why they chose the tech, and why they implemented their own metrics without consultations.
    • Can be frustrating to have expertise available to them that they don't take advantage of
    • Deb in semi-PM role is trying to help them with this
    • Discovery dashboard has all the vertical stuff in one place, consistent UI, etc. There are also valid reasons for having grafana.
  • Was hoping that after we updated the graphs, they would work for that team. Yuri wants direct immediately control over dashboard changes, which they get with grafana.
    • Not clear that grafana will be able to produce output that will be useful for demonstrating KPIs to management and other stakeholders
  • How big a project is grafana support?
    • Kind of a big project
    • Feels like a personal slight to not use our dashboards, but learning grafana would be a useful skill
  • Even if the interactive team might not appreciate the maps dashboards, other people will
    • The Discovery dashboards need to track the work of the whole team
  • Concern that Yuri might try to figure out KPIs and metrics in isolation
    • Are you optimistic that analysts will be more integrated into decisions about metrics and KPIs?
      • Some optimism. There is a ticket out there.
      • Analysis team tries to be proactive and forward-looking; interactive team tends to be more quick with less forward planning

Action items

edit
  • Mikhail: Discuss opsy stuff w/GL (and Chelsy)
  • Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
    • Include a statement of the perceived problem