Core Platform Team/Initiative/API Gateway/Documentation Plan

View the prototype Track this project on Phabricator Share feedback Subscribe to project updates

Overview

The API Gateway is a CPT initiative to create a rate-limited API proxy at api.wikimedia.org as part of the Platform Evolution program. The MVP scope includes the MediaWiki REST API (including page, media, history, and search endpoints) and the Feeds API, with the intention of adding APIs over time. See the initiative page and Phabricator tag.

Approximate list of endpoints for MVP

GET /feed/v1/{project}/{language}/announcements GET /feed/v1/{project}/{language}/featured/{yyyy}/{mm}/{dd} GET /feed/v1/{project}/{language}/onthisday/{type}/{mm}/{dd} GET /core/v1/{project}/{language}/file/{title} GET /core/v1/{project}/{language}/search/page GET /core/v1/{project}/{language}/search/title POST /core/v1/{project}/{language}/page GET /core/v1/{project}/{language}/page/{title} PUT /core/v1/{project}/{language}/page/{title} GET /core/v1/{project}/{language}/page/{title}/bare GET /core/v1/{project}/{language}/page/{title}/html GET /core/v1/{project}/{language}/page/{title}/with_html GET /core/v1/{project}/{language}/page/{title}/links/language GET /core/v1/{project}/{language}/page/{title}/links/media GET /core/v1/{project}/{language}/page/{title}/history GET /core/v1/{project}/{language}/page/{title}/history/counts/{type} GET /core/v1/{project}/{language}/revision/{id}/bare GET /core/v1/{project}/{language}/revision/{from}/compare/{to}

The purpose of this document is to manage the planning and feedback process for the documentation for this project.

Components

Wikimedia API Gateway architecture diagram

MediaWiki

The API Portal will be a MediaWiki instance at api.wikimedia.org (beta: api.wikimedia.beta.wmflabs.org)

WikimediaApiPortal skin (Phabricator project)

A MediaWiki skin providing the look-and-feel of the API Portal

WikimediaApiPortalOAuth extension (Phabricator project)

A MediaWiki extension running on the API Portal that interfaces with the OAuth extension on Meta-Wiki

OAuth extension on Meta-Wiki

Responsible for storing and managing OAuth clients

Implementation plan

ID	Description	Owner	Status
Phase 1: Planning and development
1	Finalize development contract	Cindy	Done
2	Security approval for the Chameleon skin (phab:T246949)	Cindy	Cancelled (See phab:T252462.)
3	Collect feedback from Technical Engagement	Alex	Done (Notes)
4	Create a wiki instance phab:T246945 (New Public Wiki for the API Portal) phab:T246946 (Prepare and check storage layer)	Cindy	Done
5	Submit request to create a beta wiki instance (phab:T254185)	Cindy	Done
6	Create a MediaWiki skin ("WikimediaApiPortal") that matches the provided design, and create documentation for the extension on mediawiki.org Repository Skin:WikimediaApiPortal phab:T251279 (Create API Portal skin)	Cindy	Done
7	Define technical specification for interactions between the API Portal and the OAuth extension phab:T247570 (Provide specification of protocol interactions initiating from API Portal) phab:T249781 (Define "Create app" flow) phab:T249783 (Define "Manage apps" flow)	Bill	Done
8	Update OAuth extension to support configuration via web API, including documentation updates phab:T257982 (Update the OAuth extension to support the API Portal)	Bill	Done
9	Create a MediaWiki extension ("WikimediaApiPortalOAuth") that provides a simplified user interface for managing OAuth 2.0 consumers through the API Portal, matching the provided design, and update documentation Repository Extension:WikimediaApiPortalOAuth phab:T251283 (Create extension to provide functionality for creating and managing OAuth 2.0 clients from API Portal)	Cindy: UI Bill: API	Done
10	Draft documentation for the API portal	Alex	Done
Phase 2: Beta testing
Prerequisites: WikimediaApiPortal skin Done, WikimediaApiPortalOAuth extension Done, and updates to OAuth extension Done are code complete
11	Deploy WikimediaApiPortal (phab:T257959) and WikimediaApiPortalOAuth to api.wikimedia.beta.wmflabs.org	Cindy	Done
12	Enable feature flag for new endpoints in the OAuth extension on beta	Cindy	Done
13	Configure and test permissions on beta site (phab:T249834)	Alex	Done
14	Test WikimediaApiPortal skin on beta site	Alex	Done
15	Test WikimediaApiPortalOAuth extension and interactions with OAuth extension on beta site with beta-Meta	Technical: Bill Design: Alex	Done
Phase 3: Private launch
Prerequisites: Completion of Phase 2 Viable Envoy configuration in Kubernetes (See phab:T246945#6274493) Security reviews for WikimediaApiPortal skin (phab:T246949) Done, WikimediaApiPortalOAuth extension (phab:T254947) Done, and updates to OAuth extension (phab:T254948) Done are complete
16	Deploy WikimediaApiPortal and WikimediaApiPortalOAuth to api.wikimedia.org	Cindy	Done
17	Configure permissions on api.wikimedia.org to enable private launch (Homepage and login page are publicly visible; other content is private) (phab:T249834)	Cindy	Done
18	Add content to api.wikimedia.org, review, and iterate phab:T255483 (Add tabbed window gadget for code samples)	Alex	Done
Phase 4: Content testing
Prerequisites: Completion of Phase 3 Completion of phab:T235276 (Client Developer uses MediaWiki REST API) and phab:T246265 (Client Developer uses Wikifeeds API)
19	Enable feature flag in the OAuth extension to enable new functionality	Cindy	Done
20	Test tutorials and code samples	Alex	Done
Phase 5: Public launch
Prerequisites: Completion of Phase 4 Public launch of API Gateway (phab:T255032) Performance reviews for WikimediaApiPortal skin (phab:T252462) Done, WikimediaApiPortalOAuth extension (phab:T254950) Done, and updates to OAuth extension (phab:T254951) Done are complete
21	Update permissions to allow public read access to all content on api.wikimedia.org	Cindy	Done
22	Work with engineers to plan, draft, and review documentation for implementers of the API Gateway (phab:T251440)	Hugh	Done wikitech:API Gateway
23	Pass end-to-end testing for the Gateway, Portal, and OAuth extension in production (phab:T268257)	Alex	Done
24	Announce launch to the wider movement (phab:T268257)	Alex	Postponed following product management handoff

Documentation for end-users of the API (API Portal)

Requirements

Documentation requirements

• How to use the api.wikimedia.org API
• How to use the OAuth 2.0 flow with the api.wikimedia.org API
• Rate limiting policies
• Links to other Wikimedia APIs and resources
• Visual design similar to the prototype
• Shared Wikimedia account login

OAuth 2.0 client management requirements

• Developers can create, list, and disable OAuth 2.0 clients using a simplified interface
• Visual and interaction design similar to the prototype

Scope

MVP scope

• Manually created API reference docs and guides in English
• Editing restricted to a docs-editors group while we stabilize the content and set up contribution flows (See phab:T249834)
• User feedback via talk pages and “Was this page helpful?” gadget
• OAuth 2.0 client management interface
• Strategy for reducing duplication between the the API portal and the existing docs for the MediaWiki REST API and Feeds API.
• Pageview data for the API portal

Future scope

• API sandbox
• Translate extension
• Contribution flow for volunteers
• Automated reference docs integrated with translatewiki
• Asynchronous review process for contributors

Out of scope

• Updates to the UI of the OAuth special pages on Meta

Implementation strategy

• Use MediaWiki as the content platform
• Implement the visual design using a new MediaWiki skin, configured to look similar to the prototype. Certain items such as the behaviors/flyouts may differ since they would use styles from Bootstrap. In the future, the visual style can be implemented using the improved Vector skin.
• Create an extension that provides a simple UI to create simple OAuth 2.0 clients as shown in the prototype.
  • The new extensions will only be installed on the API Portal.
  • The new extension will interact with the OAuth extension installed on Meta through an API.
  • The new extension will be named WikimediaApiPortal and could also contain other UI elements needed for the API Portal.
• Attempt to limit any additional libraries necessary for the UI.

Naming and location

Proposed location: MediaWiki instance at:

• api.wikimedia.org (preferred)
• Alternatives:
  • developer.wikimedia.org (could be confusing due to the fact that these docs won’t have the comprehensive scope of a Wikimedia Developer Portal)
  • restapi.wikimedia.org
  • apiportal.wikimedia.org
  • webapi.wikimedia.org
  • apidev.wikimedia.org
  • apidoc.wikimedia.org
  • dev.wikimedia.org (exists as a redirect to a page on mediawiki.org)

Considerations

Due to the maintenance effort required to sustain a new public wiki, we should seek to balance the design requirements of this project with the principles and processes of the Wikimedia movement.

Open questions

•  Done Is there a way we could enable posting on talk pages without enabling editing of all pages?
  • Manual:Preventing access implies that Extension:Lockdown might be needed. --AKlapper (WMF) (talk) 17:09, 26 March 2020 (UTC)
  • (See phab:T249834)

Prototype

View the API Portal prototype here, including:

• Documentation intro page
• API reference overview
• Search pages reference
• Community page
• My account pages including create key popup

Documentation for implementers of the API Gateway and rate limiting system

Documentation requirements

• Overview of the implementation and configuration of the API Gateway and rate limiting system
• How to manage rate limits
• How to add and remove APIs from the Gateway

Implementation strategy

Proposed location: Wikitech TechOps docs

Objectives and metrics

Create a great developer experience for the Wikimedia API

Metric: Qualitative feedback from developer interviews

Make it easy for developers to get started

Metric: Percentage of visitors to the site who register an app

Metric: Percentage of registered apps with at least one API call

Make it easy for developers to build apps

Metric: Percentage of users with registered apps that have an app with more than five API calls