Wikimedia Engineering/WMF Tech Days 2013/Product and Data Workshop

Product & Data workshop

Notes: http://etherpad.wikimedia.org/p/data-and-product-workshop

see the brainstorming page

Leads

Dario Taraborelli, Aaron Halfaker, Toby Negrin

Schedule

1:30 pm - 60 minutes - 6th floor collab space

Audience

• PMs and analysts: how do we incorporate data into product decisions?
• Engineers who want to collect data on new features, changes to core etc.

Outcomes

• how we incorporate data into product decisions
• how to work with analysts to evaluate products
• review features currently in design/under development and the most appropriate evaluation approach

Things to discuss

• Evaluating new features/products
  • Product lifecycle
  • Using data to inform product design vs product optimization
  • Types of request: counting things vs testing hypotheses
• Controlled tests (hypothesis testing)
  • When is it appropriate to run a controlled test?
  • A/B testing: low hanging fruits
  • A/B testing: state of the art. What can we do with the existing infrastructure?
  • What are the costs associated with a controlled test?
• Descriptive usage data (counting things)
  • I want to use EventLogging to collect usage data, where do I start? How do I set up a dashboard?
  • Ad-hoc analysis: e.g. feature adoption by user segment
  • Cohort analysis: Wikimetrics
• Other product evaluation methods
  • Usability testing
  • Qualitative analysis