Analytics/Roadmap/PlanningMeetings/2012 Sept 20

Notes for the Team Analytics roadmap planning meeting for 20 Sept 2012, taken by Dave Schoonover.

Need to create a set of Data Release Processes

• Search log data release contained unacceptable data. How can we prevent this in the future?
  • Diederik on point
  • Conversation with: Team Analytics, robla, Moeller, Legal, Dario, Chris MzMcBride?, Chris Steipp?
  • Who else might have a valued/paranoid perspective? MZMcBride??
• Process for NEW datasets, as well as Smoke-check each data upload prior to public notice
• Concrete threads:
  • For any new datastream: "What's the Attack Surface?"
    • Need to spend more time thinking about what sort of privacy exploits are possible
    • Strip all (no matter if we take other steps):
      • URLs? Spam, SEO links, etc
      • Email addresses
      • IP addresses
  • What criterion for k-anonymity are we going to use (if any)? --> Publish behavioral/request data only as aggregates
• Followup on this release:
  • Disclosure requirements
  • Legal is looking into the impact and our obligations
  • Need to convey clearly to the community what happened and what we're doing

Milestone Planning

By Project

Kraken

• Set up Cassandra cluster, get it working with Hadoop. [otto + dsc] (Sept)
  • Load in sample data sets. [otto] (Sept)
  • Tee the udp2log stream into Kraken. [otto + dsc] (Sept)
  • First-pass at Hive/Pig Jobs [dsc + otto] (Sept)
• Set up Maven / Somatype Artifact Repository (continuous integration) [dsc] (Oct)
  • WMF Maven parent pom [dsc] (Oct)
• Puppetize Kraken [otto] (Ongoing)
• Set up JMX monitoring -- needs to be our LAN [otto + dsc] (Oct)
• Get Storm set up [dsc + otto] (Oct)
  • Start work on ETL topology [dsc] (Oct)
• Hardware reinstallation -- Depends on Ops [otto] (Oct)
• Get to consensus with Ops regarding logging of the firehose [dsc + otto] (Oct)
  • Research needed: test running cli JVM producers does not cause extra load [otto] (Oct)

Legacy Log Collection

• Add support for new domain names in webstatscollector (blog, etc) [diederik] (Sept)
• udp2log filters
  • Update filters for Wikipedia Zero [otto] (Ongoing)
  • Filter by X-Carrier headers. [otto + asher + diederik] (Oct)
  • udp-filter to filter by http status. [otto] (Oct)

WikiStats

• Reduce backlog regarding Wikistats traffic (squid etc) scripts [stefan] (Oct)
• Repair data errors in wikistats, and add process for checking data integrity [ezachte] (Sept)
• Make wikistats more robust (MoM validations) [ezachte] (Oct)
• Add Blackbox testing to WikiStats [diederik + ezachte] (Oct)

Ops & Maintenance

• Access/support requests for stat1, stat1001 [otto] (Ongoing)
• Migrate Reportcard off Labs onto stat1001 -- reportcard.wikimedia.org [otto + dsc] (Oct)
• Maintenance of oxygen/emery/locke [otto] (Ongoing)

Data

• Publish Monthly Report Card -- deal with monthly data processing irregularities, perform correction/validation [ezachte + diederik + dsc] (Ongoing)
• Create Data Release Practices Task Force [diederik] (Sept)
• Start pushing datasets to AWS [diederik] (Oct)
• Finalize scripts to massively compact dammit.lt data [erik] (Oct)
  • Blogpost about what awesome stuff you can do with this [diederik + ?] (Oct)

Limn

• Bootstrap Dan [dan + dsc] (Sept) [DONE]
• Refactor charting to use d3 [dan + dsc]
  • Initial Prototype with Options UI (Sepåt)
  • Feature Parity with Dygraphs (plus bugfixes, etc) (Oct)
• Bugfixes (like Save-As, UI Error Notifications, ...) [dan] (Oct)
• Mirror GitHub to Gerrit [dsc] (Sept)
• Improve Limn wiki, docs, & guides (esp those shameful screenshots) [dan] (Oct)
• Coke (make for Coco) task to create symlinks into dataDir from an existing data repository (such as, say, analytics/reportcard/data) [dsc] (Sept)
  • Coke task to download and setup dummy testing data for ease of development [dsc] (Sept)
• UI support for remote datasets via proxy [dsc + dan] (Oct)
• Migrate Dario's dashboards to Limn [dsc] (Sept)
• Support the Global Dev dashboard [evan] (Ongoing)
• Support the Gerrit Stats dashboard [diederik] (Ongoing)
• Deploy reportcard / gerrit-stats on stat1001 (aka, "the Debian packaging discussion") [otto + dsc] (Oct)

By Month

September

• (Kraken) Set up Cassandra cluster, get it working with Hadoop. [otto + dsc] (Sept)
  • Load in sample data sets. [otto] (Sept)
  • Tee the udp2log stream into Kraken. [otto + dsc] (Sept)
  • First-pass at Hive/Pig Jobs [dsc + otto] (Sept)
• (Kraken) Puppetize Kraken [otto] (Ongoing)
• (Legacy Log Collection) Add support for new domain names in webstatscollector (blog, etc) [diederik] (Sept)
• (Data) Create Data Release Practices Task Force [diederik] (Sept)
• (Limn) Bootstrap Dan [dan + dsc] (Sept) [DONE]
• (Limn) Refactor charting to use d3 [dan + dsc]
  • Initial Prototype with Options UI (Sept)
• (Limn) Mirror GitHub to Gerrit [dsc] (Sept)
• (Limn) Coke (make for Coco) task to create symlinks into dataDir from an existing data repository (such as, say, analytics/reportcard/data) [dsc] (Sept)
  • Coke task to download and setup dummy testing data for ease of development [dsc] (Sept)
• (Limn) Migrate Dario's dashboards to Limn [dsc] (Sept)
• (Limn) Support the Global Dev dashboard [evan] (ongoing)
• (Limn) Support the Gerrit Stats dashboard [diederik] (Ongoing)

October

• (Kraken) Set up Maven / Somatype Artifact Repository (continuous integration) [dsc] (Oct)
  • WMF Maven parent pom [dsc] (Oct)
• (Kraken) Puppetize Kraken [otto] (Ongoing)
• (Kraken) Set up JMX monitoring -- needs to be our LAN [otto + dsc] (Oct)
• (Kraken) Get Storm set up [dsc + otto] (Oct)
  • Start work on ETL topology [dsc] (Oct)
• (Kraken) Hardware reinstallation -- Depends on Ops [otto] (Oct)
• (Kraken) Get to consensus with Ops regarding logging of the firehose [dsc + otto] (Oct)
  • Research needed: test running cli JVM producers does not cause extra load [otto] (Oct)
• (Legacy Log Collection) udp2log filters
  • Update filters for Wikipedia Zero [otto] (Ongoing)
  • Filter by X-Carrier headers. [otto + asher + diederik] (Oct)
  • udp-filter to filter by http status. [otto] (Oct)
• (WikiStats) Reduce backlog regarding Wikistats traffic (squid etc) scripts [stefan] (Oct)
• (WikiStats) Make wikistats more robust (MoM validations) [ezachte] (Oct)
• (WikiStats) Add Blackbox testing to WikiStats [diederik + ezachte] (Oct)
• (Ops & Maintenance) Access/support requests for stat1, stat1001 [otto] (Ongoing)
• (Ops & Maintenance) Migrate Reportcard off Labs onto stat1001 -- reportcard.wikimedia.org [otto + dsc] (Oct)
• (Ops & Maintenance) Maintenance of oxygen/emery/locke [otto] (Ongoing)
• (Data) Publish Monthly Report Card -- deal with monthly data processing irregularities, perform correction/validation [ezachte + diederik + dsc] (Ongoing)
• (Data) Start pushing datasets to AWS [diederik] (Oct)
• (Data) Finalize scripts to massively compact dammit.lt data [erik] (Oct)
  • Blogpost about what awesome stuff you can do with this [diederik + ?] (Oct)
• (Limn) Refactor charting to use d3 [dan + dsc]
  • Feature Parity with Dygraphs (plus bugfixes, etc) (Oct)
• (Limn) Bugfixes (like Save-As, UI Error Notifications, ...) [dan] (Oct)
• (Limn) Improve Limn wiki, docs, & guides (esp those shameful screenshots) [dan] (Oct)
• (Limn) UI support for remote datasets via proxy [dsc + dan] (Oct)
• (Limn) Support the Global Dev dashboard [evan] (Ongoing)
• (Limn) Support the Gerrit Stats dashboard [diederik] (Ongoing)
• (Limn) Deploy reportcard / gerrit-stats on stat1001 (aka, "the Debian packaging discussion") [otto + dsc] (Oct)

Followups

• [dsc] Update wiki with project pages for everything on the Roadmap page
  • Each project owner will then update their Project Status for Sept
• [dsc] Update the Engineering Roadmap wiki page: https://www.mediawiki.org/wiki/Roadmap
• [dsc] Fill in week-by-week team roadmap without breakout by project

&heart;

<3 http://art.less.ly/2012/heart-dino.png <3