Wikimedia Technology/Annual Plans/FY2019/TEC5: Scoring Platform/Goals

Program Goals and Status for FY18/19Edit

  • Goal Owner: Aaron Halfaker
  • Program Goals for FY18/19: Improve the efficiency of wiki processes and mitigate the effects of algorithmic biases that are introduced.
  • Annual Plan: TEC5 Scoring Platform
    • Primary Goal is Knowledge Equity: Grow new contributors and content
    • Tech Goal: Supporting our Community of contributors

Edit

Outcome 1 / Output 1Edit

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE team

Goal(s)Edit

  • Keep ORES online and improve robustness

StatusEdit

  Note: July 30, 2018

  In progress

  Note: August 9, 2018

  In progress

  Note: September 13, 2018

  In progress...and will be forever and ever :)


Outcome 2 / Output 3Edit

Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)

Published posts about ORES, AI, wiki processes, etc. in the ORES manual

Dependencies on: Cloud Services team

Goal(s)Edit

  • Documentation -- Threshold optimizations in The ORES Manual

StatusEdit

  Note: July 30, 2018

  In progress

  Note: August 9, 2018

  In progress

  Note: September 13, 2018

  In progress and is in review with Sarah R


Outcome 2 / Output 4Edit

Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)

Workshops run, papers published, datasets published, tutorials published, hackathons co-organized

Goal(s)Edit

  • Developing a focus group for JADE

StatusEdit

  Note: July 30, 2018

  In progress

  Note: August 9, 2018

  In progress

  Note: September 13, 2018

  In progress


Outcome 3 / Output 5Edit

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Dependencies on: SRE Team

Goal(s)Edit

  • JADE --> Production

StatusEdit

  Note: July 30, 2018

 N Blocked The block on our JADE --> Production goal is a serious threat at this point, but there's nothing to do except to follow an RFC in the short term.

  Note: August 9, 2018

 N Blocked DBA's have confirmed a ceiling of growth for JADE in MediaWiki. An RFC discussion is scheduled for Aug. 22nd

  Note: September 13, 2018

Currently still  N Blocked because TechCom has requested iteration on our secondary tables for MediaWiki before they will consider closing the discussion

Edit

Outcome 1 / Output 1Edit

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE, Security teams

Goal(s)Edit

  • Upgrade Celery to version 4   Done
  • Fix logging (redirect useful logs to logstash)   Done
  • Implement edit quality models for translatewiki    In progress should be done by end of quarter

StatusEdit

  Note: October 18, 2018

  To do

  Note: December 13, 2018

These goals are   Partially done at this point and will be fully   Done by end of quarter.


Outcome 2 / Output 3Edit

Grow the community of wiki decision process modelers and tool builders (staff, volunteers, academics)

Published posts about ORES, AI, wiki processes, etc. on the Wikimedia blog in ORES manual

Dependencies on: Cloud Services Team

Goal(s)Edit

  • Documentation -- Feature Injection in The ORES Manual   In progress
  • Blog announcement of the Undisclosed Paid Editors dataset   In progress
  • Resubmit ORES paper to the Journal of Social Computing  N Cancelled

StatusEdit

  Note: October 18, 2018

  To do

  Note: December 13, 2018

This is   In progress and will be finished by end of this quarter; although we will not resubmit the paper.


Outcome 3 / Output 5Edit

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Dependencies on: SRE Team

Goal(s)Edit

  • JADE --> Production (4 small wiki deployments pending local consensus)

StatusEdit

  Note: October 18, 2018

  In progress

  Note: December 13, 2018

This is   Partially done and will be finished in early Q3

Edit

Outcome 3 / Output 5Edit

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Goal(s)Edit

  • Implement basic Jade integration (Phab:T183381) (carryover + additional commitment)

StatusEdit

  Note: January 10, 2019

  • Discussed that as we've just gotten back from our vacations, this work is ramping up and is   In progress

  Note: February 14, 2019

  • Working with design on the production of wireframes/mocks   In progress

  Note: March 18, 2019

  • Stalled due to a loss in Engineering Capacity  N Not done


Outcome 1 / Output 1Edit

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE, Security teams

Goal(s)Edit

  • Prepare ORES for kubernetes (ETA: 2 quarters)
  • Implement high-availability strategy for redis
  • Implement safe task serialization for celery (JSON)
  • Implement new models in response to demand
  • Perform research in response to community/WMF needs

StatusEdit

  Note: January 10, 2019

  • Discussed that as we've just gotten back from our vacations, this work is ramping up and is   In progress

  Note: February 14, 2019

  • New models for Italian, Korean, and German Wikipedia as well as Spanish Wikibooks.
  • Nearly finished with conversion from Pickle to JSON in Celery
  • The Teahouse AI-Hostbot study has started
  • Kubernetes work is   In progress

  Note: March 18, 2019

  • Large scale deployments of February achievements (new/improved models, JSON for celery, etc.)
  • Kubernetes work continues (as planned, this will take multiple quarters)
    • Wikilabels CI is now running fully on docker


Strategy/Annual planningEdit

Engage with 3-5 year strategy and annual planning process. Dependencies on: Audiences, Research, Search

GoalsEdit

  • Develop proposal for building "topic spaces" infrastructure
  • Develop annual plan program proposal for FY20

StatusEdit

  Note: January 10, 2019

  • Discussed that as we've just gotten back from our vacations, this work is ramping up and is   In progress

  Note: February 14, 2019

  • Topic spaces proposal has been socialized and a document is   In progress

  Note: March 18, 2019

  • Topic spaces proposal is complete and has been socialized. Talks with Audiences started   Done
  • Annual plan work in progress   In progress

Edit

Outcome 1 / Output 1Edit

More wiki communities benefit from semi-automated curation support

ORES supports the edit quality prediction models for more wikis/languages

Dependencies on: SRE

Goal(s)Edit

  • Prepare ORES for kubernetes (ETA: 2 quarters)
  • Implement high-availability strategy for redis
  • Improve monitoring of ORES components in grafana
  • Implement new models in response to demand
  • Perform research in response to community/WMF needs

StatusEdit

  Note: May 9, 2019

  • ORES for kubernetes is  N Stalled for this quarter
  • Implement high-availability strategy for redis is currently  N Blocked on figuring out the correct solution to take
  • Improve monitoring of ORES components in grafana is   Done
  • Implement new models is   In progress and we've completed svwiki article quality (model built -- pending deploy   Partially done) already; srwiki goodfaith (model improved) is   Done; eswikiversity (modeling   In progress); dewiki model is   Done but also  N Blocked waiting on Audiences / Growth team for RCFilters deployment.
  • Perform research in response to community/WMF needs - srwiki, ruwiki, hewiki now have low quality "goodfaith" models   Partially done for now. Additional work will still be   In progress this quarter.
  • Also   In progress is the work written up here.

  Note: June 20, 2019

  • ORES for kubernetes is  N Stalled for this quarter
  • Implement high-availability strategy for redis is currently  N Blocked on figuring out the correct solution to take
  • Implement new models is   In progress
    • eswikiversity   Done
    • jawiki   Done
    • zhwiki   Done
    • nlwiki improvements   Done
  • Perform research in response to community/WMF needs

Outcome 3 / Output 5Edit

Users of ORES-based-tools can build a repository of human judgement to contrast with model-predictions

JADE (our auditing system) accepts and stores human judgements

Dependencies on: Design/Audiences

Goal(s)Edit

  • Complete Jade design work. Prepare for implementation once Engineering capacity is restored.

StatusEdit

  Note: May 9, 2019

  • This design work is   In progress but progressing very slowly with reviews of the wireframes.

  Note: June 20, 2019

  • This design work is  N Postponed

Misc GoalsEdit

  • Hiring a Senior SWE and possibly an Engineering Manager
  • Interviewing is   In progress for the Sr SWE and Eng Mgr roles.
  •   Done Hired ACraze
  • Annual planning work: Propose growth model for ORES and related AI technology teams
  • This is now   Done in regards to annual planning.