User:CFloyd (WMF)/Reading Services Coordination

Purpose of this document

This document will serve as a stage for initial planning and information gathering for an initiative to improve our services processes in the Reading vertical.

Goal

To provide support and processes that enable product teams within the Reading vertical to better plan and develop new features that require backend services.

Desired Outcomes

• Reduce uncertainty of feature planning that involves services
• Improve productivity of engineers by streamlining services planning and development
• Increase collaboration of platform teams within Reading by working together on shared services
• Establish collaborative relationships with other teams and verticals that develop and maintain services

Roles and Responsibilities

To accomplish these goals, it is proposed to create a new role within the Reading vertical which will have the following responsibilities:

• Assist PMs in productizing existing and new services developed within the WMF and community
• Coordinate and communicate service requirements of the product in the Reading vertical with external teams
• Gather service needs for upcoming features from Reading PMs
• Align Reading product teams on shipping of common service backed features
• Maintain a 6-9 month road map for service dependent Reading features
• Coordinate development and deployment schedules of services and product features among teams
• Identify and manage development dependencies of service and product teams
• Establish a Point-of-Contact for each team involved in the development each service
• Maintain a Phabricator project and board for tracking schedules and dependencies

Q2 Priorities

1. Ship Donation API for the feed to support donations in apps
2. Ship Trending API for real time trending articles in the feed
3. Ship On This Day API for getting Anniversary data into the feed
4. Document how to format main page content for feed inclusion
5. Establish norms for shipping a service from inception to platform deployment
6. Establish a framework for resourcing service development in Reading
   1. Establish metrics for improving the recommendation service
7. Document requirements for offline, PWA, shared article presentation for all mobile platforms
8. Document requirements for a platform agnostic push notification service
9. Document requirements for reading platform ORES features

Services and Features

Ongoing

Feed Services APIs

Team	POC	Deliverables
Reading	Bernd	End points for each micro service End point for aggregate service
Services	?	?

Description

Create and deploy APIs and services to support the Explore Feed for the iOS and Android Apps

1. Deploy separate endpoints for random, trending, picture of the day, featured article, and in the news
2. Create an aggregated endpoint
3. Create an infrastructure that allows us to easily add new service endpoints for the feed

Maps and Geo Services API

Team	POC	Deliverables
Reading	Bernd	End points for each micro service End point for aggregate service

Description

Enhance the existing search APIs to allow:

1. Specifying a keyword search term
2. Increasing the search radius

Trending Edits API

Team	POC	Deliverables
Reading	Jon Robson	End point for service Algorithm for determining trending
Services	Gabriel Wick	Production environment for service Persistance for edit data

Description

Develop and deploy a service that will provide a list of trending articles

1. Ingest the RCS or other stream
2. Provide a means to persist and aggregate the edit information for calculations
3. Calculate trending articles based on an algorithm that analyzes number/velocity of edits
4. Provide the list in order of most to least trending
5. Provide the list in chronological order of most recently to least recently became trending

On This Day API

Team	POC	Deliverables
Reading	Monte	End point for service
Reading	Bernd	Deployment

Develop and deploy a service to provide content form the On This day and Day pages

1. Parse "On This Day" for the current day
2. Parse Date Page for the current day
3. Create a JSON object with sections for important, holidays, births, deaths, events
4. Update Feed endpoint to return this data for the corresponding day

Upcoming

Push Notifications

Team	POC	Deliverables
Reading	Gabriel Wick	Infrastructure for push notification services

Description

Create and deploy a backend infrastructure that can:

1. Send push notifications to Android, iOS, Chrome, Firefox, and Safari.
2. Can consume an arbitrary stream or end point to get data
3. Can perform arbitrary analysis of consumed data to determining whether a push is required and for what devices
4. Can persist a list of UDIDs representing devices/browsers, subscription types, dates, and some metadata
5. SLA of 10 minutes (not real time)

Trending Push Notifications

Team	POC	Deliverables
Reading	?	?
Services	?	?

Description

Send push notifications to devices and browsers when an article becomes a "large" trend

Wikidata Enrichment APIs

Team	POC	Deliverables
Discovery	?	?

Description

Ingest Wikidata information to enrich the querying ability of search APIs

Key Value Store

Team	POC	Deliverables
Reading Infrastructure	Brad Jorsch	?

Description

A generic key-value store for arbitrary user data

https://phabricator.wikimedia.org/T128602

Consolidation of CSS and JS transforms for article display

Description

Move as much styling code and JS into the MCS as possibly in order to unify the styling code for the Apps

Team	POC	Deliverables
Reading	MHolloway	?

Improve Read More Service

Team	POC	Deliverables
Research	Ellery Wulczyn?	API
Reading	?	MCS API

Description

https://phabricator.wikimedia.org/T142009

Thumb API

Team	POC	Deliverables
Editing	Brion	Architecture Approval
Services	Gabriel	Thumbnail API
?	?	Updated Article Response

Description

Create an API to return thumbnails based on defined size parameters

https://phabricator.wikimedia.org/T66214

MCS Page View enhancement

Team	POC	Deliverables
Reading	?	Article API enhancement
Reading	?	Trending API enhancement

Description

Add page view data to both the article response and the trending API response to give better context to the articles to enable UI like:

http://top.hatnote.com

On This Day API

Team	POC	Deliverables
Reading	?	On this Day API

Description

Add an API for querying OTD and full day results. This will enable platforms to display timeline cards.

Example local implementation:

https://github.com/wikimedia/wikipedia-ios/pull/701

MCS Page Content API enhancement

Team	POC	Deliverables
Reading	?	Article API enhancement
Reading	?	Trending API enhancement

Description

Two goals:

1. Improve rendering time
2. Work on ways to enable more code to be moved to the service layer and shared (CSS/JS/HTML)
3. Better cache utilization - sharing cache for all platforms

Potentially look into returning responses as HTML (instead of JSON)

API Documentation consolidation

Research and setup a way to get all API documentation for both PHP and RESTBase all in "one place"

Goals:

1. Automatic
2. Comprehensive
3. Public
4. Easy for developers to do

Administrative

• Define responsibilities for my work in establishing a services coordination process
• Create a Wiki to track progress and plans during initial stages
• Work with Adam to gather organization information and contacts
• Set up a standing meeting with Bernd to coordinate upcoming features with current MCS work
• Schedule meeting with Toby/Adam to discus current plans
• Meet with Max (TPG) to review current notes and plans to get ideas on how to orchestrate this work
• Review existing Q1 plans of Reading, Discover, Services, and Analytics
• Review existing Q2 plans of Reading, Discover, Services, and Analytics
• Establish norms for setting metrics for determining the performance of endpoints
• Establish and document norms for resourcing service development within reading
• Make sure goals for MCS are aligned with current work
• Review planned feature releases for all Reading platforms
• Understand and document the relationship between MediaWiki, Restbase Services and the Mobile Content Service
• Determine the process for planning services adoption for Mobile Web / Web (Specifically codify that the testing grounds will be apps and the Wb will go through longer trials)
• Create 12 month road map of services development
• Create 6-9 Month Road Map of all Reading Services including deployment schedules per platform
• Work with TPG to establish a process for working with teams to develop and ship cross vertical features
• Create a Phabricator tag and board to track tasks across all dependencies and platforms
• Create documentation for planning a feature with service requirements

iOS Team Impacts

1. Current time being spent on coordination is about 10-15 hours/week

Progress Reports

June 20 - June 30

• Information Gathering

July 8

• Worked with Josh to coordinate new Geodata service changes with Discovery - enhancements are deployed

July 11

• Coordinated beginning of work on new Trending Edits Service

July 18

• Created this wiki to document planning process

Aug 3

• Scheduled standing meeting with Bernd
• Added new services to track… improved read more, consolidation of article styling code, thumbnail API, On this day, and HTML
• Scheduled meeting with Max to review planning

Aug 18

• Gathered and documented requirements for iOS and Android for the forthcoming trending API
• Scheduled meeting to set the schedule for delivering trending edits APIs

Aug 24

• Bump thread on testing new More like API

Aug 26

• Add on this day API
• Work on Q2 goals with Bernd
• Start conversation for iOS resources to be used to develop a service
• Start conversation on how to track reading resources being used for service development

Sep 2

• Ensure time for developing the trending service can be allocated by Reading Web

Sep 13

• Establish and communicate norms for resourcing services development within reading
• Create road map for MCS

Sep 26

• Setup ticket for understanding saved page usage in apps
• Help with offline working group with and make plans for adoption within Reading platforms
• Send email to establish a new metric for which to evaluate the new Read more service

Oct 5

• Get requiremnts for EN donation endpoint for the feed https://phabricator.wikimedia.org/T145830

Oct 10

• Gather product requirement for Offline and PWA

Oct 17

• Follow up on Read more metrics
• Schedule development of EN donation endpoint for early November
• Schedule development of On This Day endpoint for Early November
• Follow up on scheduling development of Trending API with Olga
• Scheduled meeting with Dario to discus getting Research work into the products