User:CFloyd (WMF)/Reading Services Coordination

Purpose of this document

This document will serve as a stage for initial planning and information gathering for an initiative to improve our services processes in the Reading vertical.

Goal

To provide support and processes that enable product teams within the Reading vertical to better plan and develop new features that require backend services.

Desired Outcomes

Reduce uncertainty of feature planning that involves services
Improve productivity of engineers by streamlining services planning and development
Increase collaboration of platform teams within Reading by working together on shared services
Establish collaborative relationships with other teams and verticals that develop and maintain services

Roles and Responsibilities

To accomplish these goals, it is proposed to create a new role within the Reading vertical which will have the following responsibilities:

Assist PMs in productizing existing and new services developed within the WMF and community
Coordinate and communicate service requirements of the product in the Reading vertical with external teams
Gather service needs for upcoming features from Reading PMs
Align Reading product teams on shipping of common service backed features
Maintain a 6-9 month road map for service dependent Reading features
Coordinate development and deployment schedules of services and product features among teams
Identify and manage development dependencies of service and product teams
Establish a Point-of-Contact for each team involved in the development each service
Maintain a Phabricator project and board for tracking schedules and dependencies

Q2 Priorities

Ship Donation API for the feed to support donations in apps
Ship Trending API for real time trending articles in the feed
Ship On This Day API for getting Anniversary data into the feed
Document how to format main page content for feed inclusion
Establish norms for shipping a service from inception to platform deployment
Establish a framework for resourcing service development in Reading
1. Establish metrics for improving the recommendation service
Document requirements for offline, PWA, shared article presentation for all mobile platforms
Document requirements for a platform agnostic push notification service
Document requirements for reading platform ORES features

Services and Features

Ongoing

Feed Services APIs

Team	POC	Deliverables
Reading	Bernd	End points for each micro service End point for aggregate service
Services	?	?

Description

Create and deploy APIs and services to support the Explore Feed for the iOS and Android Apps

Deploy separate endpoints for random, trending, picture of the day, featured article, and in the news
Create an aggregated endpoint
Create an infrastructure that allows us to easily add new service endpoints for the feed

Maps and Geo Services API

Team	POC	Deliverables
Reading	Bernd	End points for each micro service End point for aggregate service

Description

Enhance the existing search APIs to allow:

Specifying a keyword search term
Increasing the search radius

Trending Edits API

Team	POC	Deliverables
Reading	Jon Robson	End point for service Algorithm for determining trending
Services	Gabriel Wick	Production environment for service Persistance for edit data

Description

Develop and deploy a service that will provide a list of trending articles

Ingest the RCS or other stream
Provide a means to persist and aggregate the edit information for calculations
Calculate trending articles based on an algorithm that analyzes number/velocity of edits
Provide the list in order of most to least trending
Provide the list in chronological order of most recently to least recently became trending

On This Day API

Team	POC	Deliverables
Reading	Monte	End point for service
Reading	Bernd	Deployment

Develop and deploy a service to provide content form the On This day and Day pages

Parse "On This Day" for the current day
Parse Date Page for the current day
Create a JSON object with sections for important, holidays, births, deaths, events
Update Feed endpoint to return this data for the corresponding day

Upcoming

Push Notifications

Team	POC	Deliverables
Reading	Gabriel Wick	Infrastructure for push notification services

Description

Create and deploy a backend infrastructure that can:

Send push notifications to Android, iOS, Chrome, Firefox, and Safari.
Can consume an arbitrary stream or end point to get data
Can perform arbitrary analysis of consumed data to determining whether a push is required and for what devices
Can persist a list of UDIDs representing devices/browsers, subscription types, dates, and some metadata
SLA of 10 minutes (not real time)

Trending Push Notifications

Team	POC	Deliverables
Reading	?	?
Services	?	?

Description

Send push notifications to devices and browsers when an article becomes a "large" trend

Wikidata Enrichment APIs

Team	POC	Deliverables
Discovery	?	?

Description

Ingest Wikidata information to enrich the querying ability of search APIs

Key Value Store

Team	POC	Deliverables
Reading Infrastructure	Brad Jorsch	?

Description

A generic key-value store for arbitrary user data

https://phabricator.wikimedia.org/T128602

Consolidation of CSS and JS transforms for article display

Description

Move as much styling code and JS into the MCS as possibly in order to unify the styling code for the Apps

Team	POC	Deliverables
Reading	MHolloway	?

Improve Read More Service

Team	POC	Deliverables
Research	Ellery Wulczyn?	API
Reading	?	MCS API

Description

https://phabricator.wikimedia.org/T142009

Thumb API

Team	POC	Deliverables
Editing	Brion	Architecture Approval
Services	Gabriel	Thumbnail API
?	?	Updated Article Response

Description

Create an API to return thumbnails based on defined size parameters

https://phabricator.wikimedia.org/T66214

MCS Page View enhancement

Team	POC	Deliverables
Reading	?	Article API enhancement
Reading	?	Trending API enhancement

Description

Add page view data to both the article response and the trending API response to give better context to the articles to enable UI like:

http://top.hatnote.com

On This Day API

Team	POC	Deliverables
Reading	?	On this Day API

Description

Add an API for querying OTD and full day results. This will enable platforms to display timeline cards.

Example local implementation:

https://github.com/wikimedia/wikipedia-ios/pull/701

MCS Page Content API enhancement

Team	POC	Deliverables
Reading	?	Article API enhancement
Reading	?	Trending API enhancement

Description

Two goals:

Improve rendering time
Work on ways to enable more code to be moved to the service layer and shared (CSS/JS/HTML)
Better cache utilization - sharing cache for all platforms

Potentially look into returning responses as HTML (instead of JSON)

API Documentation consolidation

Research and setup a way to get all API documentation for both PHP and RESTBase all in "one place"

Goals:

Automatic
Comprehensive
Public
Easy for developers to do

Administrative

~~Define responsibilities for my work in establishing a services coordination process~~
~~Create a Wiki to track progress and plans during initial stages~~
~~Work with Adam to gather organization information and contacts~~
~~Set up a standing meeting with Bernd to coordinate upcoming features with current MCS work~~
~~Schedule meeting with Toby/Adam to discus current plans~~
~~Meet with Max (TPG) to review current notes and plans to get ideas on how to orchestrate this work~~
~~Review existing Q1 plans of Reading, Discover, Services, and Analytics~~
Review existing Q2 plans of Reading, Discover, Services, and Analytics
Establish norms for setting metrics for determining the performance of endpoints
Establish and document norms for resourcing service development within reading
~~Make sure goals for MCS are aligned with current work~~
~~Review planned feature releases for all Reading platforms~~
Understand and document the relationship between MediaWiki, Restbase Services and the Mobile Content Service
Determine the process for planning services adoption for Mobile Web / Web (Specifically codify that the testing grounds will be apps and the Wb will go through longer trials)
~~Create 12 month road map of services development~~
Create 6-9 Month Road Map of all Reading Services including deployment schedules per platform
Work with TPG to establish a process for working with teams to develop and ship cross vertical features
Create a Phabricator tag and board to track tasks across all dependencies and platforms
Create documentation for planning a feature with service requirements

iOS Team Impacts

Current time being spent on coordination is about 10-15 hours/week

Progress Reports

June 20 - June 30

Information Gathering

July 8

Worked with Josh to coordinate new Geodata service changes with Discovery - enhancements are deployed

July 11

Coordinated beginning of work on new Trending Edits Service

July 18

Created this wiki to document planning process

Aug 3

Scheduled standing meeting with Bernd
Added new services to track… improved read more, consolidation of article styling code, thumbnail API, On this day, and HTML
Scheduled meeting with Max to review planning

Aug 18

Gathered and documented requirements for iOS and Android for the forthcoming trending API
Scheduled meeting to set the schedule for delivering trending edits APIs

Aug 24

Bump thread on testing new More like API

Aug 26

Add on this day API
Work on Q2 goals with Bernd
Start conversation for iOS resources to be used to develop a service
Start conversation on how to track reading resources being used for service development

Sep 2

Ensure time for developing the trending service can be allocated by Reading Web

Sep 13

Establish and communicate norms for resourcing services development within reading
Create road map for MCS

Sep 26

Setup ticket for understanding saved page usage in apps
Help with offline working group with and make plans for adoption within Reading platforms
Send email to establish a new metric for which to evaluate the new Read more service

Oct 5

Get requiremnts for EN donation endpoint for the feed https://phabricator.wikimedia.org/T145830

Oct 10

Gather product requirement for Offline and PWA

Oct 17

Follow up on Read more metrics
Schedule development of EN donation endpoint for early November
Schedule development of On This Day endpoint for Early November
Follow up on scheduling development of Trending API with Olga
Scheduled meeting with Dario to discus getting Research work into the products