Requests for comment/Extension management with Composer/Farms and Composer

From mediawiki.org

This page is a summary of the various issues and solutions about managing MediaWiki extensions with Composer in a farm.

This page comes for the topic "A way to disable extensions" and from personal (Seb35) research I did for the extension MediaWikiFarm (see references at the end). It should evolve with the state of the art.

Definitions[edit]

  • Extension: a MediaWiki extension or skin
  • Composer-managed Extension: Extension installed with the Composer mechanism: adding it in composer.json and running Composer
  • (Extension) Activation:
    Preliminary, Composer’s initial goal is to manage libraries, which one can define as passive code (e.g. classes or functions: no main() function = if you don’t explicitely call it, it will never be executed and hence will have absolutely no effect – other than reserve of symbol names). This corresponds to class autoloading (MediaWiki class autoloader and Composer PSR4/PSR0/classmap/files when the files only contains classes or functions definitions)
    MediaWiki extensions are generally active code (they register Hooks (term for the various MediaWiki mechanisms like hooks, namespaces registration, AuthManager mechanisms, permissions regitration, etc.) providing features). The extension’s code becomes active when registered in MediaWiki, this can be through:
    1. Legacy activation: require_once the extension, registering the Hooks (possibly indirectly by a function/method)
    2. Composer activation: autoloading a file registering the Hooks (possibly indirectly by a function/method)
    3. ExtensionRegistry: (since MediaWiki 1.25) direct registration of the Hooks (possibly indirectly by a callback function/method)
  • Wiki: a MediaWiki website, with a dedicated database and URL, and a specific configuration and (Activated) extensions set
  • Wiki farm: a set of Wikis

Constraints[edit]

  • Dependencies: Composer-managed MediaWiki extensions have dependencies in their composer.json, these MUST be handled.
  • Single code: in a farm, you typically want to have only one (or a few) MediaWiki codebase (say, 1.27 LTS) to reduce maintenance burden and possibly CPU (Composer computations) and disk space (MW 1.27 = 110 Mio + vendor = 30 Mio), and activate only some extensions depending on the wiki (by the way, it’s what Wikimedia does, the biggest MediaWiki farm)

Issue[edit]

You want a Wiki farm with different Wikis, each one with a specific configuration and extensions set.

Consequently, given some MediaWiki extensions are Composer-managed, you want to Activate only some Extensions, and hence find all Dependencies for Activated Extensions.

Solutions[edit]

Each solution has its advantages and drawbacks. They are sorted by increasing difficulty, the more you scroll down the non-standard it becomes and the more you need carefully-crafted tools.

1. One MediaWiki installation per Wiki[edit]

Details

Each Wiki has a dedicated directly containing a MediaWiki installation with the specific directory corresponding to the Activated Extensions.

Advantages
  • Each Wiki is completely independent from the others, you can have one LocalSettings per directory or, if you prefer, one central LocalSettings, and there is one composer.json and one vendor directory per Wiki, this is the ultra-classical installation
  • Each Wiki can have completely different versions for non-Composer-managed Extensions and for Composer-managed Extensions
Drawbacks
  • When you want to activate/deactivate an Extension, you have to check if it is Composer-managed, and if so, run Composer to recompute the vendor directory
  • When you want to change the MediaWiki version globally, you have to use some batch script
  • The size is linear with the number of Wikis (currently about 140 Mio + Extensions, multiplied by the number of Wikis), the CPU is linear with the number of Wikis

2. One MediaWiki installation with one vendor directory per Wiki[edit]

Details

All Wikis use the same directory containing a unique MediaWiki installation, but each vendor directory is computed for the specific Wiki, and the file vendor/autoload.php contains some logic to include the Composer autoloader corresponding to the Wiki. Obviously, you should have a specific handling of the LocalSettings and images, cache, etc directories, but this is a farm topic unrelated to Composer (same remark for next solutions).

Advantages
  • Only one MediaWiki installation (= constant size for MediaWiki codebase and Extensions)
  • When you want to change the MediaWiki version globally, you can prepare a unique MediaWiki installation, and put it in production when ready
  • Each Wiki can have completely different versions for Composer-managed Extensions
Drawbacks
  • Non-standard setup, you have to wrap Composer in some script to create a specific vendor directory in a non-standard path
  • When you want to activate/deactivate an Extension, you have to check if it is Composer-managed, and if so, run the specific-flavour Composer to recompute the vendor directory corresponding to the Wiki
  • When you want to change the MediaWiki version globally, you have to use some batch script to compute the specific vendor directories
  • Wikis have the very same versions for non-Composer-managed Extensions

3. One MediaWiki installation with one vendor directory containing one composer subdirectory per Wiki[edit]

Details

All Wikis use the same directory containing a unique MediaWiki installation and all Wikis use the same vendor directory containing classical Composer-managed librairies, Composer-managed Extensions have their code installed with Composer (in the extensions or skins MediaWiki directories); Composer is run as much time as there are combinations of Composer-managed Extensions, and each autoloading set contained in a vendor/composer directory is moved in a vendor/composerKEY directory, and the file vendor/autoload.php contains some logic to include the Composer autoloader corresponding to the Wiki.

The specific vendor/composerKEY directories can be prepared beforehand (scenario A) or when a specific Composer-managed Extensions set is requested (scenario B). All small improvement would be to create symbolic links for identical combinations (when an Extension requires other Extensions).

Advantages
  • (A) You can instantly activate the Composer-managed Extensions
  • When you want to change the MediaWiki version globally, you can prepare a unique MediaWiki installation, and put it in production when ready
  • (B) Very few resources are needed
Drawbacks
  • Non-standard setup, you have to wrap Composer in some script to create a specific vendor/composerKEY directories
  • (A) Exponential number of combinations of Composer-managed Extensions = 2^N (see Wikipedia); disk size is not limiting (500 Kio per combination), but CPU time will become limiting, and it is quite a waste of resources only a few combinations are used
  • You have to prepare the various vendor/composerKEY directories beforehand
  • Wikis have the very same versions for non-Composer-managed Extensions and Composer-managed Extensions

4. One MediaWiki installation with one vendor directory containing one composer subdirectory per Extension[edit]

Details

Similar to the previous setup, but instead of computing all combinations, you compute only the Composer autoloader for each Composer-managed Extension (idependently of other Composer-managed Extensions). In the file vendor/autoload.php, you include all Extension’s autoloaders corresponding to Activated Extensions: you will load as much Composer autoloaders as there are Activated Extensions (or a bit less if you know some Extensions are – automatically – Activated by others).

Possibly a class can be autoloaded by multiple Composer autoloaders, but it is not important, particularly given the corresponding file is always the same. Note that recently (2016), some Extensions (PageForms, SemanticFormsSelect) are Composer-installed but ExtensionRegistry-Activated, so the Activation should correspond to the Composer installation to avoid "all Composer-installed Extensions correctly installed but some Dependencies of Activated Extension are not Activated" (more clearly SemanticFormsSelect Activated but PageForms (Composer Dependency) not Activated).

Advantages
  • You can instantly activate the Composer-managed Extensions
  • When you want to change the MediaWiki version globally, you can prepare a unique MediaWiki installation, and put it in production when ready
  • Very few resources are needed: linear (and small) with the number of Extensions and independent of the number of Wikis
Drawbacks
  • Non-standard setup, you have to wrap Composer in some script to create a specific vendor/composerKEY directories
  • You have to prepare the various vendor/composerKEY directories beforehand
  • Wikis have the very same versions for non-Composer-managed Extensions and Composer-managed Extensions

5. One MediaWiki installation with one vendor directory with a modified autoloader of files, to activate files corresponding to Activated Extensions[edit]

Details

On Phabricator T61872#802235 detailled by User:Egel, install Composer in the traditional way, then remove the autoloading of the files (=the entry points of the extensions are always in the files; the other autoloading methods are only for passive code (classes and functions)). Then, extensions are Activated by including the "autoloading files" corresponding to the Activated Extensions. A further improvement would be to run Composer as much times there are Composer-managed Extensions to find Dependencies between Extensions, and next use these informations to Activate coherent sets of Extensions.

Advantages
  • You can instantly activate the Composer-managed Extensions
  • When you want to change the MediaWiki version globally, you can prepare a unique MediaWiki installation, and put it in production when ready
  • Very very few resources are needed: independent of the number Wikis
Drawbacks
  • Non-standard setup, you have to wrap Composer in some script to create the specific vendor/composer/autoload_files_KEY.php files
  • You have to prepare the various vendor/composer/autoload_files_KEY.php files beforehand
  • Wikis have the very same versions for non-Composer-managed Extensions and Composer-managed Extensions

Composer-based[edit]

To be researched. Possibly create a Composer plugin implementing one of the previous strategy or another approach.

Implementations[edit]

References[edit]

Retrieved from "https://www.mediawiki.org/w/index.php?title=Requests_for_comment/Extension_management_with_Composer/Farms_and_Composer&oldid=3809722"