Wikimedia Product/Better use of data/DACI

During Q1 FY 2018-2019 (July 2018 - September 2018) the Better Use of Data Working Group and interested Wikimedia Audiences team members collaborated on the following DACI to help better formalize the instrumentation (usually taken to mean event logging) process. This artifact is part of Output 3.1 Instrumentation.

Activity Comments Head of Product Analytics Data Analyst / Data Scientist Reading Infrastructure Engineering Manager Reading Infrastructure Data Engineer Product Software Engineer Software Engineer Research Analytics Engineering Engineering Manager Analytics Engineering Software Engineer Product Manager / Program Manager / Researcher / Designer Legal / Privacy Security Management (usually, product Director)
Decide metrics needs Usually the product manager or researcher realizes a need for metrics and gets the OK from management. Sometimes, management was the one initiating the request in the first place. The workflow is initiated. Contributor Informed Informed Driver
Define research questions, metrics, instrumentation data The product manager defines questions that the instrumentation data needs to be able to help us answer.

The product manager and the data analyst determine the concrete metrics to be derived from the instrumentation, and define the concrete data that the instrumentation willl need to generate for this purpose. The data engineer and product software engineer reality check the instrumentation definition to ensure the data can actually be generated with the event model of the client.

Approver Contributor Contributor Contributor Driver
Instrumentation specification ticket and schema blob (precise definition of schema fields w/ validation, data dictionary cross-check, sampling approach, purging & whitelisting approach, queries, metadata) The schema and queries and other attributes of the logging and use are collaboratively defined in Phabricator and a schema blob through a session between the data analyst, product software engineer, AE software engineer (optional), and data engineer.

The data analyst creates database queries that will be used to generate results (this is the equivalent of test-driven development (TDD), for data). Once they're done, the Heaad of Product Analytics reviews regarding use of best practices with the Product Manager and then advances to the next step for privacy review. Everyone else is looped onto the ticket. A heads up of 7-10 business days is given to Legal/Privacy via privacy@ that schema privacy review will be needed on date X.

Approver Contributor Informed Contributor Contributor Informed Contributor Driver Informed Informed Informed
Schema privacy review privacy@ is emailed with the link to the ticket the business day before (or the business morning of) date X for the review so that Legal/Privacy can review and approve the formulation or ask for re-engineering. Contributor Driver Approver Contributor
Schema creation with backlink to ticket The data analyst, data engineer, and product engineer finalize the schema, with a reference back to the Phabricator ticket. Ideally, all metadata is self-documented in the schema / schema registry itself. The Head of Product Analytics reviews with the Product Manager and confirms when the schema is satisfactory for being instrumented against. Approver Contributor Contributor Contributor Driver
Instrumentation, pipeline scripts (if needed), alerting & monitoring Product Software Engineer does the plumbing, with assistance from the data engineer as needed. The data analyst reviews and approves the code/config artifacts and testing is arranged. Approver Contributor Contributor Informed Driver
Testing The software engineers (usually, product software engineers), the data analyst and possibly a QA tester collaboratively test the instrumentation. The data analyst reviews the outcome with the product manager and if the work is satisfactory move forward. Approver Contributor Contributor Contributor Driver
Update data dictionary if needed The data dictionary (initially on-wiki, probably built into the schema registry as a software fixture when the schema registry is built) is updated by the data analyst Approver Contributor Contributor Contributor Contributor Driver
Activation The product software engineer or data analyst activates production level logging. Contributor Contributor Contributor Approver Driver
Verification and fixes Analytics Engineering Software Engineer confirms no system degradation. Approver Driver
Dashboard productionization In case it has been requested to visualize some of the instrumentation data in form of a dashboard, the data analyst configures dashboarding, consulting with the data engineer and product software and Analytics Engineering software engineer as needed. Driver Contributor Contributor Contributor Approver
Long-term Support (LTS) The data analyst monitors the quality of the logged data and reports anomalies/bugs to the Reading Infrastructure Engineering Manager and Product Software Engineer. Driver Informed Contributor Informed Informed Approver
Decommissioning If logging is no longer needed, the Head of Product Analytics requests sign off on decommission. The data analyst and software engineers provide consultation and work along the way to actually decommission things. Driver Contributor Informed Contributor Contributor Informed Contributor Contributor Contributor Informed Approver