Wikimedia Technology/Annual Plans/FY2019/TEC5: Scoring Platform/Goals

Program Goals and Status for FY18/19

TEC5: Scoring Platform (ORES)

• Goal Owner: Aaron Halfaker
• Program Goals for FY18/19: Improve the efficiency of wiki processes and mitigate the effects of algorithmic biases that are introduced.
• Annual Plan: TEC5 Scoring Platform
  • Primary Goal is Knowledge Equity: Grow new contributors and content
  • Tech Goal: Supporting our Community of contributors

Q1 Goals

Outcome 1 / Output 1

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE team

Goal(s)

• Keep ORES online and improve robustness

Status

 Note: July 30, 2018

In progress

 Note: August 9, 2018

In progress

 Note: September 13, 2018

In progress...and will be forever and ever :)

Outcome 2 / Output 3

Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)

Published posts about ORES, AI, wiki processes, etc. in the ORES manual

Dependencies on: Cloud Services team

Goal(s)

• Documentation -- Threshold optimizations in The ORES Manual

Status

 Note: July 30, 2018

In progress

 Note: August 9, 2018

In progress

 Note: September 13, 2018

In progress and is in review with Sarah R

Outcome 2 / Output 4

Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)

Workshops run, papers published, datasets published, tutorials published, hackathons co-organized

Goal(s)

• Developing a focus group for JADE

Status

 Note: July 30, 2018

In progress

 Note: August 9, 2018

In progress

 Note: September 13, 2018

In progress

Outcome 3 / Output 5

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Dependencies on: SRE Team

Goal(s)

• JADE --> Production

Status

 Note: July 30, 2018

N Blocked The block on our JADE --> Production goal is a serious threat at this point, but there's nothing to do except to follow an RFC in the short term.

 Note: August 9, 2018

N Blocked DBA's have confirmed a ceiling of growth for JADE in MediaWiki. An RFC discussion is scheduled for Aug. 22nd

 Note: September 13, 2018

Currently still N Blocked because TechCom has requested iteration on our secondary tables for MediaWiki before they will consider closing the discussion

Q2 Goals

Outcome 1 / Output 1

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE, Security teams

Goal(s)

• Upgrade Celery to version 4  Done
• Fix logging (redirect useful logs to logstash)  Done
• Implement edit quality models for translatewiki   In progress should be done by end of quarter

Status

 Note: October 18, 2018

To do

 Note: December 13, 2018

These goals are Partially done at this point and will be fully  Done by end of quarter.

Outcome 2 / Output 3

Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)

Published posts about ORES, AI, wiki processes, etc. on the Wikimedia blog in ORES manual

Dependencies on: Cloud Services Team

Goal(s)

• Documentation -- Feature Injection in The ORES Manual In progress
• Blog announcement of the Undisclosed Paid Editors dataset In progress
• Resubmit ORES paper to the Journal of Social Computing N Cancelled

Status

 Note: October 18, 2018

To do

 Note: December 13, 2018

This is In progress and will be finished by end of this quarter; although we will not resubmit the paper.

Outcome 3 / Output 5

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Dependencies on: SRE Team

Goal(s)

• JADE --> Production (4 small wiki deployments pending local consensus)

Status

 Note: October 18, 2018

In progress

 Note: December 13, 2018

This is Partially done and will be finished in early Q3

Q3 Goals

Outcome 3 / Output 5

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Goal(s)

• Implement basic Jade integration (Phab:T183381) (carryover + additional commitment)

Status

 Note: January 10, 2019

• Discussed that as we've just gotten back from our vacations, this work is ramping up and is In progress

 Note: February 14, 2019

• Working with design on the production of wireframes/mocks In progress

 Note: March 18, 2019

• Stalled due to a loss in Engineering Capacity N Not done

Outcome 1 / Output 1

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE, Security teams

Goal(s)

• Prepare ORES for kubernetes (ETA: 2 quarters)
• Implement high-availability strategy for redis
• Implement safe task serialization for celery (JSON)
• Implement new models in response to demand
• Perform research in response to community/WMF needs

Status

 Note: January 10, 2019

• Discussed that as we've just gotten back from our vacations, this work is ramping up and is In progress

 Note: February 14, 2019

• New models for Italian, Korean, and German Wikipedia as well as Spanish Wikibooks.
• Nearly finished with conversion from Pickle to JSON in Celery
• The Teahouse AI-Hostbot study has started
• Kubernetes work is In progress

 Note: March 18, 2019

• Large scale deployments of February achievements (new/improved models, JSON for celery, etc.)
• Kubernetes work continues (as planned, this will take multiple quarters)
  • Wikilabels CI is now running fully on docker

Strategy/Annual planning

Engage with 3-5 year strategy and annual planning process. Dependencies on: Audiences, Research, Search

Goals

• Develop proposal for building "topic spaces" infrastructure
• Develop annual plan program proposal for FY20

Status

 Note: January 10, 2019

• Discussed that as we've just gotten back from our vacations, this work is ramping up and is In progress

 Note: February 14, 2019

• Topic spaces proposal has been socialized and a document is In progress

 Note: March 18, 2019

• Topic spaces proposal is complete and has been socialized. Talks with Audiences started  Done
• Annual plan work in progress In progress

Q4 Goals

Outcome 1 / Output 1

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE

Goal(s)

• Prepare ORES for kubernetes (ETA: 2 quarters)
• Implement high-availability strategy for redis
• Improve monitoring of ORES components in grafana
• Implement new models in response to demand
• Perform research in response to community/WMF needs

Status

 Note: May 9, 2019

• ORES for kubernetes is N Stalled for this quarter
• Implement high-availability strategy for redis is currently N Blocked on figuring out the correct solution to take
• Improve monitoring of ORES components in grafana is  Done
• Implement new models is In progress and we've completed svwiki article quality (model built -- pending deploy Partially done) already; srwiki goodfaith (model improved) is  Done; eswikiversity (modeling In progress); dewiki model is  Done but also N Blocked waiting on Audiences / Growth team for RCFilters deployment.
• Perform research in response to community/WMF needs - srwiki, ruwiki, hewiki now have low quality "goodfaith" models Partially done for now. Additional work will still be In progress this quarter.
• Also In progress is the work written up here.

 Note: June 20, 2019

• ORES for kubernetes is N Stalled for this quarter
• Implement high-availability strategy for redis is currently N Blocked on figuring out the correct solution to take
• Implement new models is In progress
  • eswikiversity  Done
  • jawiki  Done
  • zhwiki  Done
  • nlwiki improvements  Done
• Perform research in response to community/WMF needs
  • m:Research:Exploring systematic bias in ORES is still Partially done (continue into next quarter)

Outcome 3 / Output 5

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Dependencies on: Design/Audiences

Goal(s)

• Complete Jade design work. Prepare for implementation once Engineering capacity is restored.

Status

 Note: May 9, 2019

• This design work is In progress but progressing very slowly with reviews of the wireframes.

 Note: June 20, 2019

• This design work is N Postponed

Misc Goals

• Hiring a Senior SWE and possibly an Engineering Manager

• Interviewing is In progress for the Sr SWE and Eng Mgr roles.
•  Done Hired ACraze

• Annual planning work: Propose growth model for ORES and related AI technology teams

• This is now  Done in regards to annual planning.