Requests for comment/Improving extension management

This RfC proposes a system for easier extension management through:
 * 1) Allowing extensions to specify their compatibility and dependencies in a standard and machine-readable method.
 * 2) Providing an API to expose said metadata
 * 3) Building a web interface that allows system administrators to search for extensions, install them, and enable/disable them.

Pain points for developers

 * Handling dependencies
 * Depending upon another extension is typically manual and hacky with to avoid load order dependencies.
 * Versioning
 * Versioning is mainly an internal thing, not really exposed anywhere, and not really used by sysadmins. Or is it? As a dev, I typically ignore them unless someone asks me to bump it.
 * Compatibility with core
 * We have REL_ branches, but people usually don't backport every change they make that is compatible
 * Some extension maintainers keep compatibility with older versions of core, but that isn't documented anywhere in a machine readable format for tools like ExtensionDistributor.
 * wfUseMW is hacky as well and until recently required a $wgExtensionFunctions.

Pain points for sysadmins

 * Just because an extension has a REL1_xx branch doesn't mean it actually is compatible with said version
 * Just because an extension has a REL1_xx branch doesn't mean master isn't compatible with an older stable release
 * Upgrading tons of extensions at once (during a major version bump) is not super friendly and requires a lot of manual steps

Considerations

 * Security: This system should function in an environment where webservers cannot write executable PHP files
 * Ease of use: Shouldn't require a CLI interface, nearly everything should be doable from the web
 * But...for people who like CLI/git, we should still support that workflow, probably with a maintenance script.

Dependencies upon MediaWiki core
Extensions can currently specify that they depend upon a specific version of core by using, however that is extremely hacky, and only allows for specifying a lower bound of support. We automatically create REL1_XX branches whenever core is branched, except that there is no actual verification that the extension works with that core version unless the maintainer or a user does so.

Instead of that, I propose that we store the versions that the extension supports in its extension.json file:

Supports is an array of versions that the maintainer has explicitly stated work and are supported. Experimental is used for those are that are in active development and not yet recommended for use. Broken means that it is not compatible with those versions. When an extension drops support for a certain release, it would create the REL1_XX branch at the last commit that did support that release.

The extension distributor would be updated to read from extension.json to see which versions to display.

Dependencies upon libraries
We have standardized upon using composer as the dependency manager for library dependencies. Extensions would list their depdencies upon external libraries through composer.json files.

For users using the web interface and without CLI access, we would have a web proxy to composer that would take an input of the required dependencies and output a tarball of the vendor directory they need. I've written a quick prototype to demonstrate how this might work.

CLI users would be able to use the composer-merge-plugin to manage library dependencies.

Dependencies upon extensions
Many times extension dependencies simply dependencies upon common code, that really should be refactored out into a separate library. A good example of this is AbuseFilter depending upon AntiSpoof. AbuseFilter really only needs AntiSpoof's normalization functionality, not the entire extension. That common code should be refactored out into a separate library which both AbuseFilter and AntiSpoof depend upon.

However, there are some cases where extensions do actually depend upon each other. These constraints can be added in the extension.json file:

Extensions that use versioning are encouraged to adopt semantic versioning, and extensions that depend upon those can use version identifiers like "1.*". For extensions that don't use versioning, "*" will get the latest version of the extension that supports your MediaWiki core version. Any operator that composer supports will be usable.

This will be processed by the registry at compile time, and exceptions will be thrown if dependencies are missing or incompatible.

Exposing information via an API
Once this information is in extension.json, we can expose it via an API. Independently I was working on improving our current ExtensionDistributor setup, and think that including this metadata with that would make sense. See Extension:ExtensionDistributor/tardist for more details.

Implementation
A web interface would exist to list installed extensions (both disabled and enabled), and would allow enabling and disabling of those extensions. It would also show whether those extensions have updates available. This page should probably be independent of MediaWiki (mw-config/) so it is accessible in case an extension breaks everything.

Sysadmins would be able to update an extension, which would download an updated tarball from extension distributor, unzip it, and install it. The composer.json would be read if it exists and dependencies will be calculated and fetched by accessing the composer functions directly.

For sysadmins who are more familiar with CLI and would prefer to use that, the same functionality would be available via a maintenance script.

Rather than writing our own dependency manager, we can re-use composer's by adding "composer/composer" as a dependency and interacting with the DependencyResolver part. (Legoktm's quick POC)

Why not use composer directly?
Regarding Requests for comment/Extension management with Composer:

As mentioned earlier, composer is what we use to manage dependencies upon libraries. It's important to note that extensions are not libraries. Libraries typically just require an autoloader, whereas extensions need an autoloader, but also need to register hooks, special pages, API modules, and much more.


 * composer is CLI-only. For shared hosting users or users who don't like CLI, we need some kind of web interface.
 * Note that https://github.com/CurosMJ/NoConsoleComposer exists, also see stackoverflow. We can also use the composer code directly and skip the CLI utility.
 * composer combines the installation process (downloading tarball and unzipping it to the right place) and the enable process (require_once/wfLoadExtension) into one step. This is especially problematic for wiki-farm setups where extensions may be enabled on one wiki but not on others (see A way to disable extensions)
 * We could just have composer download and install the extension, and *not* autoload the entry point.
 * composer is still an alpha product, and not packaged by any major distros. This means users need to download a phar file and trust it.
 * If we add composer/composer as a dependency to mediawiki/core (that would be shipped with the tarball), we wouldn't have to require the binary and could directly interact with the composer API.
 * What Dantman said:,.

All that said, composer's dependency resolver is pretty powerful, and makes sense for us to re-use (and not reinvent), but we can do that without having to use the rest of composer and package our extensions as composer dependencies.