Requests for comment/API roadmap

Background
MediaWiki API has been steadily growing and adding features, and even though it provides most of the desired functionality, I feel it is necessary to discuss our future plans for growth, versioning, and overall development strategy.

Proposals
api2.php ? agent=MyProgramVer42 & action=query~2 & ...

api2.php
Setting a new versioned entry point allows us
 * new format for warnings
 * possibly overall output
 * possibly less tolerance for incorrect requests.
 * require the new 'agent' parameter for all clients containing a meaningful string that would help us contact the broken client's author.

smart continue
Easy/smart continue allows significant client simplification. Easy continue is an API guarantee to the client that by simply adding all items in the 'continue' section to the next query the client will receive all available data, without accidentally skipping some values due to a 'limit' parameters or the generator paging. This change can be made available in all query versions, and made default in action=query~2. A client library could have this code (Python with requests lib):

query incomplete pages
Notify client if not all properties have finished populating the 'page' element, and the client should merge it with the result of the subsequent api call. E.g. action=query~2&titles=Page1|Page2&prop=links could get this result, in which 'Page2' does not have all containing links, and should be merged with the result of the next call. This change can be made for all query versions. A client could merge pages with this logic

API Cleanup

 * request rewriting (aliasing)
 * facility for renaming modules and parameters, or any other parameter manipulations.


 * 'flags' parameter should replace all the boolean flags
 * iilocalonly should be replaced with iiflags=localonly|..., etc


 * all query extensions should use 3+ letter prefixes to avoid conflicts with the core

Defaults

 * query~2 would always use the smart continue unless the client sets 'legacycontinue' parameter

Modules to Deprecate

 * list=watchlistraw
 * This module changes the output formatter, breaking the action=query data model (multiple data items no longer can be requested in one call). On top of that, it does not seem to have any benefit compared with list=watchlist.

Justification

 * Allows clients to avoid updating every time API changes
 * Reduces the cost of making a breaking change
 * Organize feature changes - if the client asks for ver X, API guarantees the capabilities of X and result in format X.
 * Recommended API usage is shown as the latest version. If API default behavior (v1) changes to be optional (v2), new developers will work with the new default (recommended) way from the start.
 * No clutter with ever expanding list of additional parameters - with versioning, new parameters could replace old ones, or change their meaning, or be removed completely without breaking any clients.
 * Ability to obsolete capabilities in a structured way: MW supports API requests with version X+, but will give standard warning for anything below the latest version Y. No need to parse warning messages to see if specific feature change applies.

Requirements
API versioning must solve these real life scenarios:
 * Client must identify itself to the host in order for us to notify developer of incorrect/suboptimal usage.
 * Client relies on the specific output format, and needs to always get the same.
 * Client wants to use feature X. How does it check if it is available.
 * Client has to be notified that feature Y is obsolete (Unsure of this)
 * Updates to the core must not change API output and behavior, except the obsolete notification
 * An extension may add functionality to the API, and might be updated independently from the core.

Per-module versioning
api.php ? action=query~2 & ... Each API module will have its own versioning number. This approach provides independence between different components, and solves the version conflict problem described in the other section.

Global Version Number
api2.php ? action=query & ...

One global version is good to deal with overall compatibility issues, such as changing the result data format: structured warnings, json only format, etc.

Despite this, even though having one main version parameter approach seems more reasonable at a first glance, it seems to be a dead end with respect to the modular expandable nature of our API. Any extension can add an API module, but no one can ever guarantee that an extension will be upgraded at the same time as the core.

Let's say there is a server with the core ver 1102 and an extension ver 1206. Later MW 1309 is released, with the new API behavior. Independently an extension 1309 is also released to support the new features.

On the server, admins update the core right away, but decide to delay extension deployment because they might want to do additional testing, or for any other reason. So now the core 1309 is working with extension 1206. This is totally fine so far - the core frequently runs with the older extensions.

After MW update, but before extension is put in production (which could be a long time), API client is created against the API version 1309. The developers use one global version number to access API - 1309 - to test and develop the new program, and they assume that the extension results will stay the same - they might not know that the new extension is pending. But later, at a whim of an admin, the extension is finally added to production, at which point the client fails - what was suppose to be a firm contract - "you ask for ver X you get ver X despite upgrades" has been broken.

To avoid any kind of surprises like this, there should never be any "default" fallback behavior in the API. The developer may query which versions are available, and choose to work with the one available, but will do so explicitly. See Defaults section below.