Project Canasta/Infrastructure solution

From mediawiki.org
Jump to navigation Jump to search

This is an informal RFC/RFP for finding/creating a system to generate downloadables for the project known as Project Canasta. Project Canasta, in brief, is aimed at enabling easy-installed MediaWiki instances that contain not just MediaWiki but also skins and extension, and a set of wiki pages, that together can fulfill some use case. Part of that task involves finding, or creating, a solution to generate these downloadables.

The solution should meet as many as possible of the following criteria:

(The person generating the downloadable is referred to here as a "creator", and the person downloading it is referred as an "administrator".)

  • The solution should let the creator take an existing MediaWiki instance (already generated - automatically generating this "master" instance, if it is going to be automatically generated, is a separate component of Project Canasta), and then generate a downloadable file or image from that instance. That file/image, when downloaded, should allow an administrator to simply run some command, and then, with a minimum of options presented, the script should generate an entire MediaWiki instance, as well as a database that is set up and initialized for both MediaWiki and all the installed extensions. This should include having a database user and password that are generated - perhaps in real time by the script - and then stored in LocalSettings.php.
  • Ideally the creator can specify whether or not this downloadable should additionally include instances of PHP, a DB system and a web server (e.g., MySQL and Apache), and potentially an operating system as well (e.g., CentOS). That way, administrators will have their choice of whether they want a container that includes all of these, or whether they want to use their own existing systems and simply download/install MediaWiki.
  • Additionally, the solution should be fully open source, and it should be relatively easy for a creator to run on their system.

Please create sections below for any existing solutions that could be used for this purpose, briefly explaining how they work and whether they could be made to fit all of these criteria.

Meza Approach[edit]

Meza logo.png
MW Documentation
Github Project

Meza is a (M)ediaWiki easy (EZ) (A)dministration program which enables the easy install and maintenance of a complete, fully-featured, ready-to-use MediaWiki application.

Prerequisites[edit]

  • To use meza all you need is a clean target system that has Centos7 (or RHEL7) installed and has access to the Internet.

Installation (Centos/RHEL)[edit]

If you already have a server that meets Meza's requirements, run the following 4 commands in your server's terminal to install Meza.

  1. sudo yum install -y git ansible
  2. sudo git clone https://github.com/enterprisemediawiki/meza /opt/meza
  3. sudo bash /opt/meza/src/scripts/getmeza.sh
  4. sudo meza deploy monolith

This will setup a demo wiki with the user Admin with password adminpass.

Visit your fully installed and pre-configured wiki at: https://localhost

What next?[edit]

Running the 4 commands above will establish a ready to use wiki-farm with 1 demo wiki pre-configured with lots of common extensions (such as VE, SMW, Page Forms, and SRF) and some standardized wiki content (like templates, properties, forms, and categories). From here you can customize the common configuration of the whole farm or just specific wikis and then put these changes into effect just by re-running the simple command: sudo meza deploy monolith.

Additionally, you can use meza to simply and quickly do many site administrator tasks such as:

  • Perform updates
  • Create new wikis (and delete existing wikis)
  • Add/Remove extensions
  • Perform backups

See: Meza/Commands and Meza#Usage for more info.

Linux/Docker: dataspectsSystemBuilder approach[edit]

The principles and concepts of the following use case steps are implemented and I use them almost every week. For now and in the context of Canasta, they shall represent my current opinion of an optimal integration/abstraction level.

Creator[edit]

The CreatorProcess (e.g. Meza, BlueSpice, dataspectsSystemBuilder, ...) shall deliver the following CreatorArtefacts components:

This represents a certain version, e.g. mediawiki-canasta-1.35.0-201215-* (see below), consisting of:

  • a specific MediaWiki core version
  • a specific set of versioned MediaWiki extensions and skins
  • a specific LocalSettings.php configuration

MediaWiki Canasta versioning ideas[edit]

  • mediawiki-canasta-1.35.0-201215-single-standard
  • mediawiki-canasta-1.35.0-201215-single-crm
  • mediawiki-canasta-1.35.0-201215-single-documentManagement
  • ...

Packer[edit]

The PackerProcess shall deliver the following PackerArtefacts in the $MEDIAWIKI_CANASTA_DISTRIBUTION_FOLDER:

  • a $MEDIAWIKI_DISTRIBUTION_ARCHIVE
  • a $MEDIAWIKI_DATABASE_DUMP in the $MEDIAWIKI_DISTRIBUTION_ARCHIVE

produced by the following bash file build-distribution-package.sh:

#!/bin/bash

MEDIAWIKI_ROOT_FOLDER=/home/dserver/mediawiki_root/w
MEDIAWIKI_CANASTA_DISTRIBUTION_FOLDER=/home/dserver
MEDIAWIKI_CANASTA_ARCHIVE=mediawiki-root-w-folder-1.35.1-3.2.2.tar.gz
MEDIAWIKI_DATABASE_DUMP=db.sql

APACHE_CONTAINER_NAME=mediawiki_canasta

MYSQL_HOST=127.0.0.1
DATABASE_NAME=mediawiki
MYSQL_USER=mediawiki
MYSQL_USER_PASSWORD=mediawikipass

# Dump database
sudo -S docker exec $APACHE_CONTAINER_NAME bash -c \
  'mysqldump -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_USER_PASSWORD \
  $DATABASE_NAME > /var/www/html/w/$MEDIAWIKI_DATABASE_DUMP'

# Pack
tar --exclude '.*' \
  -czvf $MEDIAWIKI_CANASTA_DISTRIBUTION_FOLDER/$MEDIAWIKI_CANASTA_ARCHIVE *"
  $MEDIAWIKI_ROOT_FOLDER

Installer[edit]

See https://github.com/dataspects/mediawiki-manager

Post-install setup[edit]

user@server:$SYSTEM_ROOT_FOLDER$ tree -L 3
.
├── CanastaInstanceSettings.env
├── LICENSE
├── README.md
├── docker-compose.yml
├── httpd.conf
├── install-canasta-version.sh
├── mediawiki-root-w-folder-1.35.0-3.2.1.tar.gz
├── mediawiki_root
│   └── w
│       ├── LocalSettings.php
│       ├── composer.json
│       ├── composer.local.json-sample
│       ├── composer.lock
│       ├── composer.phar
│       ├── db.sql
│       ├── extensions
│       ├── images
│       ├── install_composer.sh
│       ├── install_mediawiki_extensions.rb
│       ├── mediawiki_extensions_by_composer.yml
│       ├── mediawiki_extensions_enable_only.yml
│       ├── mediawiki_extensions_from_repo.yml
│       ├── skins
│       └── ...
└── mysql_data
    └── ...
├── ports.conf
└── switch-canasta-version.sh

Upgrader[edit]

See https://github.com/dataspects/mediawiki-manager

BlueSpice[edit]

BlueSpice is a MediaWiki distribution including >100 extensions.

Packaging[edit]

For packaging, we use composer. Extensions must be registered with a composer registry. Many MediaWiki extensions already are registered in the public composer registry, but it is also possible to maintain a separate registration.

For package definition, we use a set of composer files which are read by MediaWikis composer merge tool.

Example:

This includes the ldap stack.

{
  "repositories": [{
    "type": "composer",
    "url": "https://packages.bluespice.com/"
  }],
  "require": {
    "mediawiki/auth-remoteuser": "2.*",
    "mediawiki/ldap-authentication-2": "1.*",
    "mediawiki/ldap-authorization": "1.*",
    "mediawiki/ldap-groups": "1.*",
    "mediawiki/ldap-provider": "1.*",
    "mediawiki/ldap-user-info": "1.*" 
  }
}

(Source: https://github.com/hallowelt/mediawiki/blob/3.1.x/_bluespice/build/bluespice-ldap/composer.json)

The definitions are currently maintained in a forked repository of MediaWiki, which fetches the MediaWiki changes on a regular basis and adds the composer files into a separate folder.

The repository can be found here (current production branch): https://github.com/hallowelt/mediawiki/tree/3.1.x

Build[edit]

In order to build, we clone the repository, execute composer install and then package the folder into a tarball.

Deploy[edit]

The tarball is the basis for a Docker image, which can be found here and is fully functional out of the box:

Infrastructure Approach Capabilities Comparison[edit]

List of Capabilities Approach
# Capability Is Needed for Canasta? (Y/N) BlueSpice dataspects MW Manager MEZA Other
OS Installs tbd tbd by Docker image tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
Apache Installs tbd tbd tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
Database Installs tbd tbd tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
PHP Installs tbd tbd tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
Caching Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd
ElasticSearch Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd
Profiling Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd
Parsoid Installs tbd tbd no backward compatibility with MW<1.35 tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
NodeJS Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd
Pywikibot Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd
tbd Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd
Imagemagick Installs tbd tbd by Docker image tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
mail Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd
composer Installs tbd tbd composer.phar tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
git Installs tbd tbd no tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
Mediawiki Core Installs tbd tbd in the archive tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
Mediawiki Extensions Installs tbd tbd tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
Mediawiki Skins Installs tbd tbd tbd tbd
Updates tbd tbd tbd tbd
Configures tbd tbd tbd tbd
Synchs non-infrastructure (wiki pages) Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd
? Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd
? Installs tbd tbd tbd tbd tbd
Updates tbd tbd tbd tbd tbd
Configures tbd tbd tbd tbd tbd

Glossary[edit]

Term Definition Comments
Infrastructure
  • Anything that is not wiki-text in the wiki database or files in the wiki images folder
    Rich
  • The "infrastructure" or software stack is the software subsystems (OS, Webserver, database, PHP, etc.) that are required to run the applications (MediaWiki, ElasticSearch, etc.)
    Bryan
Wiki Content
User Content
Canasta Content
Canasta Packages
Canasta Products
Canasta Tools
MECE Mutually Exclusive Collectively Exhaustive (MECE) is a grouping principle for separating a set of items into subsets
  • Positive example: categorizing people by year of birth (assuming all years are known).
  • Negative example: categorization by nationality, because nationalities are neither mutually exclusive (some people have dual nationality) nor collectively exhaustive (some people have none).
Idempotent
  • f(f(x))=f(x)
  • In imperative programming, a subroutine with side effects is idempotent if the system state remains the same after one or several calls, in other words if the function from the system state space to itself associated to the subroutine is idempotent in the mathematical sense given in the definition;
  • In functional programming, a pure function is idempotent if it is idempotent in the mathematical sense given in the definition.