Wikimedia Discovery/Meetings/Analysis retrospective 2016-11-08
Format
editGlad/Sad/Mad: http://retrospectivewiki.org/index.php?title=Glad,_Sad,_Mad
Note that "mad" and "sad" don't have to mean literally angry or saddened. They can be used in a playful way as well.
Previous action items
edit- Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
- Include a statement of the perceived problem
- Still needs to be done
- Deb: Figure out what to do with the Maps Dashboard
- Done - fixed and updated
- Mikhail: Think about involving GL in sys admin stuff
- Need to meet w/GL
What has happened since the last retro (2016-09-30)
edit(using https://www.mediawiki.org/wiki/Discovery/Status_updates as a starting point)
- Analysis of WDQS geographic breakdown of SPARQL queries
- Wikipedia.org Portal Dashboard: investigate spike in languages visited
- Wikipedia.org dashboard: minor updates
- Bug in current data retrieval script
- Add comment to search-related eventlogging in iOS and Android apps to inform Discovery of changes
- Deb has started integrating slowly into the Discovery Interactive Team as a quasi-Product Manager
- Completed analysis of the results of BM25 AB test (final report)
- Search results page: how many visitors are on mobile vs desktop -
- Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
- Investigate recent spike in pageviews on wikipedia.org portal page
- Add a PaulScore approximation to discovery.wmflabs.org (waiting on phab:T138087 to be deployed on the train)
- Compare ZRR for query features across other search engines
- Investigate what we'd need to do to ignore double quotes in search queries
- Maps Dashboard: fix and update
- Analyze the variance of user-agent's, country, and other useful metrics of google referred traffic with and without a search query available in referrer
- Add a PaulScore approximation to discovery.wmflabs.org
What has made you mad?
edit- Interactions with Interactive team
- e.g. we created a dashboard and have made many improvements on it since its deployment, but the Interactive team does not seem to use it and even cannot remember the URL at times
- so being asked to learn a bunch of new technologies and do a bunch of additional work to make data available in a different data store so the team can do their own dashboarding is frustrating when current resource isn't being utilized
What has made you sad?
edit- Frustration with working with the Interactive team - trying to decide what is needed for metrics
- Lack of integration between Analysis and Interactive
- Chelsy and Mikhail are a resource for Interactive to use, and they are being underused
What has made you glad?
edit- Chelsy is a super fast learner and has done terrific work on her reports+1
- Coordination with Search team on test metrics/analysis of tests - quick results
- Updating the dashboards to be much better
- M & C learning new things about R and the operations stack to get new and cool technology into our environment
- Nice to be able to learn during work time
- Watching this after a tough election night: http://i.imgur.com/lZEeZ8z.gifv
- Kevin's involvement in Analysis; this team is better for having a TPG person+1+1
What has made you plaid? (aka whatever didn't fit in one of the other categories)
edit- There are several IP addresses have been doing weird things to us, e.g. pinging portal million times per day, sending the same search queries thousands of times per day. I know we are generally not blocking any of them, but wondering if we should keep a black list for ourselves, in order to remember to check them when we are doing related analysis so that we can filter out polluted data.
- Maybe a greylist rather than blacklist, just to identify for reporting?
- Publicizing the list wouldn't be appropriate (privacy issues)
- Some cases would be a judgment call; might look useless to us, but maybe they're getting value
- Maybe a greylist rather than blacklist, just to identify for reporting?
- and...this: http://i.imgur.com/sd15NVu.gifv
Discussion
editInteractive team interactions
- KS: I can see 3 areas: 1) ED-level of what metrics should be tracked, 2) tech level of whether to use grafana or other, 3) inter-team interactions and social component
- They seem to be making progress on deciding what metrics to track
- Interactive product folks (Deb/Yuri) should do some early thinking, then bring in analysts to review. Doesn't seem to be going that way so far
- For tech, there's a ticket to get data into grafana. That's frustrating because we have a dashboard system that we're really happy with, but they won't use it; want us to do additional work and learning to support this other system
- Our dashboards can do some things that grafana can't
- Unclear WHY
- Interactive team feels strongly about doing things their own way. That's why they chose the tech, and why they implemented their own metrics without consultations.
- Can be frustrating to have expertise available to them that they don't take advantage of
- Deb in semi-PM role is trying to help them with this
- Discovery dashboard has all the vertical stuff in one place, consistent UI, etc. There are also valid reasons for having grafana.
- Was hoping that after we updated the graphs, they would work for that team. Yuri wants direct immediately control over dashboard changes, which they get with grafana.
- Not clear that grafana will be able to produce output that will be useful for demonstrating KPIs to management and other stakeholders
- How big a project is grafana support?
- Kind of a big project
- Feels like a personal slight to not use our dashboards, but learning grafana would be a useful skill
- Even if the interactive team might not appreciate the maps dashboards, other people will
- The Discovery dashboards need to track the work of the whole team
- Concern that Yuri might try to figure out KPIs and metrics in isolation
- Are you optimistic that analysts will be more integrated into decisions about metrics and KPIs?
- Some optimism. There is a ticket out there.
- Analysis team tries to be proactive and forward-looking; interactive team tends to be more quick with less forward planning
- Are you optimistic that analysts will be more integrated into decisions about metrics and KPIs?
Action items
edit- Mikhail: Discuss opsy stuff w/GL (and Chelsy)
- Dan: Figure out if we want to split up Mikhail and Chelsy for meetings and teams
- Include a statement of the perceived problem