Requests for comment/API roadmap

Background
MediaWiki API has been steadily growing and adding features, and even though it provides most of the desired functionality, I feel it is necessary to discuss our future plans for growth, versioning, and overall development strategy.

Justification

 * Allows clients to avoid updating every time API changes
 * Reduces the cost of making a breaking change
 * Organize feature changes - if the client asks for ver X, API guarantees the capabilities of X and result in format X.
 * Recommended API usage is shown as the latest version. If API default behavior (v1) changes to be optional (v2), new developers will work with the new default (recommended) way from the start.
 * No clutter with ever expanding list of additional parameters - with versioning, new parameters could replace old ones, or change their meaning, or be removed completely without breaking any clients.
 * Ability to obsolete capabilities in a structured way: MW supports API requests with version X+, but will give standard warning for anything below the latest version Y. No need to parse warning messages to see if specific feature change applies.

Requirements
API versioning must solve these real life scenarios:
 * Client must identify itself to the host in order for us to notify developer of incorrect/suboptimal usage.
 * Client relies on the specific output format, and needs to always get the same.
 * Client wants to use feature X. How does it check if it is available.
 * Client has to be notified that feature Y is obsolete (Unsure of this)
 * Updates to the core must not change API output and behavior, except the obsolete notification
 * An extension may add functionality to the API, and might be updated independently from the core.

Proposals
api2.php ? agent=MyProgramVer42 & action=query~2 & ...

api2.php
Setting a new versioned entry point allows us
 * change overall output structure
 * reduce tolerance for incorrect requests
 * require the new 'agent' parameter in case the HTTP's useragent string is missing or begins with 'mozilla' or 'opera' to help us contact the broken client's author. This parameter must be part of the URL's query string even for POST requests.
 * allow new structure for warnings (See warnings and errors internationalization below)

PHP implementation [expand]

action~2 versioning
In addition to the versioned entry point, each action module could add its own versioning, which would allow:
 * remove previously added feature / parameter / behavior
 * change parameter naming
 * change parameter defaults
 * change default output format

PHP implementation [expand]

query~2 submodule versioning
Modular nature of the action=query allows us to version the individual props, lists, and meta submodules. The proposed rules are:
 * query~2 supports all nonversioned submodules that do not override default output (like watchlistraw)
 * query~2 will not allow any extension (non-core) submodules with less than 3-letter prefixes or beginning with letter 'g' (reserved for generator use).
 * a single query may not combine multiple versions of the same submodule: list=allpages|allpages~2 is invalid
 * each submodule may declare minimum query version required: list~2 may only work under query~2 or higher, but not under query.
 * the output of each submodule~n is placed under the 'query' tag without the version number: { 'query': {'submodule':...} }

Easy continue
Easy continue allows significant client simplification. Easy continue is an API guarantee to the client that by simply adding all items in the 'continue' section to the next query the client will receive all available data, without accidentally skipping some values due to a 'limit' parameters or the generator paging. This change can be made available in all query versions, and made default in action=query~2. Client logic (in python) [expand] A client library could have this code (uses python requests lib):

Query incomplete pages
Notify client if not all properties have finished populating the 'page' element, and the client should merge it with the result of the subsequent api call. E.g. action=query~2&titles=Page1|Page2&prop=links could get this result, in which 'Page2' does not have all containing links, and should be merged with the result of the next call. This change can be made for all query versions. Client logic (in python) [expand]

API Cleanup

 * request rewriting (aliasing) facility for renaming modules and parameters, or any other parameter manipulations.
 * all query extensions should use 3+ letter prefixes to avoid conflicts with the core
 * rename prop=templates into prop=transclusions - more accurate name. Prefix can stay the same.
 * list=watchlistraw should place its results under 'query' element just like all other modules.
 * list=watchlistraw</tt> and possibly list=watchlist</tt> should be renamed to clarify their meaning. watchlistraw is a list of monitored pages. watchlist is the list of changes done to the monitored pages. Looking for better naming.
 * query~2 would always use the easy continue unless the client sets 'legacycontinue' parameter
 * format=json</tt> will output pages as a list, not as a dictionary: 'pages': [ {}, {}, {} ]</tt> instead of 'pages': { '1':{}, '2':{}, '3':{} }</tt>
 * Remove indexpageids=''</tt> - won't be needed after the JSON formatter change
 * Any naming changes for the Page types table to make them more consistent?
 * Any query module that has prop</tt> parameter will always require it, when used in production (not format=xmlfm/jsonfm/...)
 * action=watch should perhaps not return ui messages that vary with the user language.
 * Add MediaWiki version with possible GIT URL(s) to meta=siteinfo</tt> (same as Special:Version page)
 * When using aplimit=max</tt>, the limits section should use parameter name, not module name: {'limits': {'aplimit': 500}}</tt> instead of {'allpages': 500}</tt>
 * Allow query to page if too many titles/pageids/revids are given. This way client does not need to worry about passing too many titles (currently api only sets a textual warning) - the query will simply treat it just like a generator, processing first N pages, and specifying that in the next query the first N values should be ignored.

Multi-writing support (Seeking comments)

 * Many wikis use multiple writing systems, and can auto-convert from one to another. Current boolean flag converttitles</tt> uses the following logic, and might need to be changed to allow for variant requests (normalize title to variant=X) or possibly other methods.

Continuation

 * Fix all modules to use continue</tt> instead of overwriting one of the original parameters
 * Both start</tt> and continue</tt> are used by AllImages</tt>, <tt>CategoryMembers</tt>, <tt>Deletedrevs</tt>, <tt>ImageInfo</tt>, <tt>UserContributions</tt>
 * <tt>from</tt> is used by <tt>AllMessages</tt>, <tt>AllUsers</tt>
 * <tt>offset</tt>	is used by <tt>ExternalLinks</tt>, <tt>ExtLinksUsage</tt>, <tt>QueryPage</tt>, <tt>Search</tt>
 * <tt>prop</tt> is used by <tt>Siteinfo</tt>
 * <tt>start</tt> is used by <tt>Blocks</tt>, <tt>LogEvents</tt>, <tt>ProtectedTitles</tt>, <tt>RecentChanges</tt>, <tt>Watchlist</tt>
 * <tt>users</tt> is used by <tt>Users</tt>

Query item count
We get lots of requests to implement count(*) functionality for various modules, and even though there is plenty of justification to get it, the fundamental database limitation has always stopped us - counting all items is an O(N) table traversal. As a result, the clients could only do a full client-side iteration of all the data and count it locally. This wastes both the server resources and bandwidth.

Now, correct me if I am wrong, but it seems that frequently the client just needs to know if the count is above a certain threshold, e.g. has a user made more than 10 edits in the last year, or does this page have more than 1 page linking to it. We could easily implement this with a <tt>count</tt> parameter: ? action=query~2 & ... & count=backlinks|links & bllimit=100 & pllimit=100 If the module name is listed in the <tt>count</tt> parameter, the resulting element is replaced with the count. I believe the api users would be happy with this compromise, and in case they really do have to know the exact count and iterate over all items, would save a lot of bandwidth. The implementation is fairly straightforward - ApiQuery.php would replace any list or prop with the module's name, and would also allow the module to optimize internal SQL.

Errors, Warnings, and Internationalization support
Mediawiki employes a very good translatewiki.net tool for all translation needs, and I think we should use that, instead of each module providing a list of warnings and error messages. We could introduce global <tt>lang=code</tt> parameter that would specify what language the user needs the message in. In case of an error or a warning, API will translate the message into the required language, or wiki's default language if <tt>lang=(nothing)</tt>. Optionally we might decide to use the HTTP Accept-Language header. In case the <tt>lang</tt> parameter is not provided (or a magic keyword 'none'? TBD), the results are returned as arrays:

Implementation details [expand] Message translation sequence for error/warning code=blbadcontinue, with params=['A','B'], and <tt>lang=ru</tt> (python style). This could be done as a method in the ApiBase, with optional override by the module subclass. The goal is to have one common repository of error messages that extensions can use, yet allow extensions to provide custom translation tables. TBD.

Modules refactoring
Here are modules that might duplicate functionality, appear closely related to be merged into one, or whose features should be moved out into a different/new module.
 * <tt>action=sitematrix</tt> extension
 * make into query submodule <tt>meta=sitematrix</tt>,
 * <tt>action=sitematrix</tt> seems to partially duplicate <tt>meta=siteinfo & siprop=interwikimap</tt>

Tokens
This section needs improvement. It will describe the API token infrastructure, both client usage and internal practices.
 * remove base::getToken - possibly replace by getTokenSalt
 * main:setupModule - can $gettoken be false?

Changes under discussion

 * per module <tt>flags</tt> parameter should replace all the boolean flags of that module:
 * query - <tt>redirects=&export=&indexpageids=''</tt> should be replaced with <tt>flags=redirects|export|indexpageids</tt>
 * imageinfo - <tt>iilocalonly</tt> should be replaced with <tt>iiflags=localonly|...</tt>, etc