User:CFloyd (WMF)/Reading Services Coordination

Purpose of this document
This document will serve as a stage for initial planning and information gathering for an initiative to improve our services processes in the Reading vertical.

Goal
To provide support and processes that enable product teams within the Reading vertical to better plan and develop new features that require backend services.

Desired Outcomes

 * Reduce uncertainty of feature planning that involves services
 * Improve productivity of engineers by streamlining services planning and development
 * Increase collaboration of platform teams within Reading by working together on shared services
 * Establish collaborative relationships with other teams and verticals that develop and maintain services

Roles and Responsibilities
To accomplish these goals, it is proposed to create a new role within the Reading vertical which will have the following responsibilities:
 * Assist PMs in productizing existing and new services developed within the WMF and community
 * Coordinate and communicate service requirements of the product in the Reading vertical with external teams
 * Gather service needs for upcoming features from Reading PMs
 * Align Reading product teams on shipping of common service backed features
 * Maintain a 6-9 month road map for service dependent Reading features
 * Coordinate development and deployment schedules of services and product features among teams
 * Identify and manage development dependencies of service and product teams
 * Establish a Point-of-Contact for each team involved in the development each service
 * Maintain a Phabricator project and board for tracking schedules and dependencies

Q2 Priorities

 * 1) Ship Donation API for the feed to support donations in apps
 * 2) Ship Trending API for real time trending articles in the feed
 * 3) Ship On This Day API for getting Anniversary data into the feed
 * 4) Document how to format main page content for feed inclusion
 * 5) Establish norms for shipping a service from inception to platform deployment
 * 6) Establish a framework for resourcing service development in Reading
 * 7) Establish metrics for improving the recommendation service
 * 8) Document requirements for offline, PWA, shared article presentation for all mobile platforms
 * 9) Document requirements for a platform agnostic push notification service
 * 10) Document requirements for reading platform ORES features

Feed Services APIs
Description

Create and deploy APIs and services to support the Explore Feed for the iOS and Android Apps
 * 1) Deploy separate endpoints for random, trending, picture of the day, featured article, and in the news
 * 2) Create an aggregated endpoint
 * 3) Create an infrastructure that allows us to easily add new service endpoints for the feed

Maps and Geo Services API
Description

Enhance the existing search APIs to allow:
 * 1) Specifying a keyword search term
 * 2) Increasing the search radius

Trending Edits API
Description

Develop and deploy a service that will provide a list of trending articles
 * 1) Ingest the RCS or other stream
 * 2) Provide a means to persist and aggregate the edit information for calculations
 * 3) Calculate trending articles based on an algorithm that analyzes number/velocity of edits
 * 4) Provide the list in order of most to least trending
 * 5) Provide the list in chronological order of most recently to least recently became trending

On This Day API
Develop and deploy a service to provide content form the On This day and Day pages
 * 1) Parse "On This Day" for the current day
 * 2) Parse Date Page for the current day
 * 3) Create a JSON object with sections for important, holidays, births, deaths, events
 * 4) Update Feed endpoint to return this data for the corresponding day

Push Notifications
Description

Create and deploy a backend infrastructure that can:
 * 1) Send push notifications to Android, iOS, Chrome, Firefox, and Safari.
 * 2) Can consume an arbitrary stream or end point to get data
 * 3) Can perform arbitrary analysis of consumed data to determining whether a push is required and for what devices
 * 4) Can persist a list of UDIDs representing devices/browsers, subscription types, dates, and some metadata
 * 5) SLA of 10 minutes (not real time)

Trending Push Notifications
Description

Send push notifications to devices and browsers when an article becomes a "large" trend

Wikidata Enrichment APIs
Description

Ingest Wikidata information to enrich the querying ability of search APIs

Key Value Store
Description

A generic key-value store for arbitrary user data

https://phabricator.wikimedia.org/T128602

Consolidation of CSS and JS transforms for article display
Description

Move as much styling code and JS into the MCS as possibly in order to unify the styling code for the Apps

Improve Read More Service
Description

https://phabricator.wikimedia.org/T142009

Thumb API
Description

Create an API to return thumbnails based on defined size parameters

https://phabricator.wikimedia.org/T66214

MCS Page View enhancement
Description

Add page view data to both the article response and the trending API response to give better context to the articles to enable UI like:

http://top.hatnote.com

On This Day API

Description

Add an API for querying OTD and full day results. This will enable platforms to display timeline cards.

Example local implementation:

https://github.com/wikimedia/wikipedia-ios/pull/701

MCS Page Content API enhancement
Description

Two goals: Potentially look into returning responses as HTML (instead of JSON)
 * 1) Improve rendering time
 * 2) Work on ways to enable more code to be moved to the service layer and shared (CSS/JS/HTML)
 * 3) Better cache utilization - sharing cache for all platforms

API Documentation consolidation
Research and setup a way to get all API documentation for both PHP and RESTBase all in "one place"

Goals:
 * 1) Automatic
 * 2) Comprehensive
 * 3) Public
 * 4) Easy for developers to do

Administrative

 * Define responsibilities for my work in establishing a services coordination process
 * Create a Wiki to track progress and plans during initial stages
 * Work with Adam to gather organization information and contacts
 * Set up a standing meeting with Bernd to coordinate upcoming features with current MCS work
 * Schedule meeting with Toby/Adam to discus current plans
 * Meet with Max (TPG) to review current notes and plans to get ideas on how to orchestrate this work
 * Review existing Q1 plans of Reading, Discover, Services, and Analytics
 * Review existing Q2 plans of Reading, Discover, Services, and Analytics
 * Establish norms for setting metrics for determining the performance of endpoints
 * Establish and document norms for resourcing service development within reading
 * Make sure goals for MCS are aligned with current work
 * Review planned feature releases for all Reading platforms
 * Understand and document the relationship between MediaWiki, Restbase Services and the Mobile Content Service
 * Determine the process for planning services adoption for Mobile Web / Web (Specifically codify that the testing grounds will be apps and the Wb will go through longer trials)
 * Create 12 month road map of services development
 * Create 6-9 Month Road Map of all Reading Services including deployment schedules per platform
 * Work with TPG to establish a process for working with teams to develop and ship cross vertical features
 * Create a Phabricator tag and board to track tasks across all dependencies and platforms
 * Create documentation for planning a feature with service requirements

iOS Team Impacts

 * 1) Current time being spent on coordination is about 10-15 hours/week

Progress Reports
June 20 - June 30 July 8 July 11 July 18 Aug 3 Aug 18 Aug 24 Aug 26 Sep 2 Sep 13 Sep 26 Oct 5 Oct 10 Oct 17
 * Information Gathering
 * Worked with Josh to coordinate new Geodata service changes with Discovery - enhancements are deployed
 * Coordinated beginning of work on new Trending Edits Service
 * Created this wiki to document planning process
 * Scheduled standing meeting with Bernd
 * Added new services to track… improved read more, consolidation of article styling code, thumbnail API, On this day, and HTML
 * Scheduled meeting with Max to review planning
 * Gathered and documented requirements for iOS and Android for the forthcoming trending API
 * Scheduled meeting to set the schedule for delivering trending edits APIs
 * Bump thread on testing new More like API
 * Add on this day API
 * Work on Q2 goals with Bernd
 * Start conversation for iOS resources to be used to develop a service
 * Start conversation on how to track reading resources being used for service development
 * Ensure time for developing the trending service can be allocated by Reading Web
 * Establish and communicate norms for resourcing services development within reading
 * Create road map for MCS
 * Setup ticket for understanding saved page usage in apps
 * Help with offline working group with and make plans for adoption within Reading platforms
 * Send email to establish a new metric for which to evaluate the new Read more service
 * Get requiremnts for EN donation endpoint for the feed https://phabricator.wikimedia.org/T145830
 * Gather product requirement for Offline and PWA
 * Follow up on Read more metrics
 * Schedule development of EN donation endpoint for early November
 * Schedule development of On This Day endpoint for Early November
 * Follow up on scheduling development of Trending API with Olga
 * Scheduled meeting with Dario to discus getting Research work into the products