Project Canasta/Infrastructure solution

This is an informal RFC/RFP for finding/creating a system to generate downloadables for the project known as Project Canasta. Project Canasta, in brief, is aimed at enabling easy-installed MediaWiki instances that contain not just MediaWiki but also skins and extension, and a set of wiki pages, that together can fulfill some use case. Part of that task involves finding, or creating, a solution to generate these downloadables.

The solution should meet as many as possible of the following criteria:

(The person generating the downloadable is referred to here as a "creator", and the person downloading it is referred as an "administrator".)

The solution should let the creator take an existing MediaWiki instance (already generated - automatically generating this "master" instance, if it is going to be automatically generated, is a separate component of Project Canasta), and then generate a downloadable file or image from that instance. That file/image, when downloaded, should allow an administrator to simply run some command, and then, with a minimum of options presented, the script should generate an entire MediaWiki instance, as well as a database that is set up and initialized for both MediaWiki and all the installed extensions. This should include having a database user and password that are generated - perhaps in real time by the script - and then stored in LocalSettings.php.

Ideally the creator can specify whether or not this downloadable should additionally include instances of PHP, a DB system and a web server (e.g., MySQL and Apache), and potentially an operating system as well (e.g., CentOS). That way, administrators will have their choice of whether they want a container that includes all of these, or whether they want to use their own existing systems and simply download/install MediaWiki.

Additionally, the solution should be fully open source, and it should be relatively easy for a creator to run on their system.

Please create sections below for any existing solutions that could be used for this purpose, briefly explaining how they work and whether they could be made to fit all of these criteria.

Meza Approach edit

MW Documentation

Github Project

Meza is a (M)ediaWiki easy (EZ) (A)dministration program which enables the easy install and maintenance of a complete, fully-featured, ready-to-use MediaWiki application.

Prerequisites edit

To use meza all you need is a clean target system that has RHEL8 (or Rocky Linux 8) installed and has access to the Internet.

Installation (RHEL/Rocky) edit

If you already have a server that meets Meza's requirements, run the following 4 commands in your server's terminal to install Meza.

sudo yum install -y git ansible
sudo git clone https://github.com/nasa/meza /opt/meza
sudo bash /opt/meza/src/scripts/getmeza.sh
sudo meza deploy monolith

This will setup a demo wiki with the user Admin with password adminpass.

Visit your fully installed and pre-configured wiki at: https://localhost

What next? edit

Running the 4 commands above will establish a ready to use wiki-farm with 1 demo wiki pre-configured with lots of common extensions (such as VE, SMW, Page Forms, and SRF) and some standardized wiki content (like templates, properties, forms, and categories). From here you can customize the common configuration of the whole farm or just specific wikis and then put these changes into effect just by re-running the simple command: sudo meza deploy monolith.

Additionally, you can use meza to simply and quickly do many site administrator tasks such as:

Perform updates
Create new wikis (and delete existing wikis)
Add/Remove extensions
Perform backups

See: Meza/Commands and Meza#Usage for more info.

Linux/Docker: dataspectsSystemBuilder approach edit

The principles and concepts of the following use case steps are implemented and I use them almost every week. For now and in the context of Canasta, they shall represent my current opinion of an optimal integration/abstraction level.

Creator edit

The CreatorProcess (e.g. Meza , BlueSpice, dataspectsSystemBuilder, ...) shall deliver the following CreatorArtefacts components:

a MediaWiki root folder $MEDIAWIKI_ROOT_FOLDER containing a certain MediaWiki Canasta version (see below)
a corresponding database $DATABASE_NAME at $MYSQL_HOST used by $MYSQL_USER identified by $MYSQL_USER_PASSWORD
a docker-compose.yml file (e.g. https://github.com/dataspects/mediawiki-manager/blob/main/docker-compose.yml)
a Dockerfile for the Apache container

This represents a certain version, e.g. mediawiki-canasta-1.35.0-201215-* (see below), consisting of:

a specific MediaWiki core version
a specific set of versioned MediaWiki extensions and skins
a specific LocalSettings.php configuration

MediaWiki Canasta versioning ideas edit

mediawiki-canasta-1.35.0-201215-single-standard
mediawiki-canasta-1.35.0-201215-single-crm
mediawiki-canasta-1.35.0-201215-single-documentManagement
...

Packer edit

The PackerProcess shall deliver the following PackerArtefacts in the $MEDIAWIKI_CANASTA_DISTRIBUTION_FOLDER:

a $MEDIAWIKI_DISTRIBUTION_ARCHIVE
a $MEDIAWIKI_DATABASE_DUMP in the $MEDIAWIKI_DISTRIBUTION_ARCHIVE

produced by the following bash file build-distribution-package.sh:

#!/bin/bash

MEDIAWIKI_ROOT_FOLDER=/home/dserver/mediawiki_root/w
MEDIAWIKI_CANASTA_DISTRIBUTION_FOLDER=/home/dserver
MEDIAWIKI_CANASTA_ARCHIVE=mediawiki-root-w-folder-1.35.1-3.2.2.tar.gz
MEDIAWIKI_DATABASE_DUMP=db.sql

APACHE_CONTAINER_NAME=mediawiki_canasta

MYSQL_HOST=127.0.0.1
DATABASE_NAME=mediawiki
MYSQL_USER=mediawiki
MYSQL_USER_PASSWORD=mediawikipass

# Dump database
sudo -S docker exec $APACHE_CONTAINER_NAME bash -c \
  'mysqldump -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_USER_PASSWORD \
  $DATABASE_NAME > /var/www/html/w/$MEDIAWIKI_DATABASE_DUMP'

# Pack
tar --exclude '.*' \
  -czvf $MEDIAWIKI_CANASTA_DISTRIBUTION_FOLDER/$MEDIAWIKI_CANASTA_ARCHIVE *"
  $MEDIAWIKI_ROOT_FOLDER

Installer edit

See https://github.com/dataspects/mediawiki-manager

Post-install setup edit

user@server:$SYSTEM_ROOT_FOLDER$ tree -L 3
.
├── CanastaInstanceSettings.env
├── LICENSE
├── README.md
├── docker-compose.yml
├── httpd.conf
├── install-canasta-version.sh
├── mediawiki-root-w-folder-1.35.0-3.2.1.tar.gz
├── mediawiki_root
│   └── w
│       ├── LocalSettings.php
│       ├── composer.json
│       ├── composer.local.json-sample
│       ├── composer.lock
│       ├── composer.phar
│       ├── db.sql
│       ├── extensions
│       ├── images
│       ├── install_composer.sh
│       ├── install_mediawiki_extensions.rb
│       ├── mediawiki_extensions_by_composer.yml
│       ├── mediawiki_extensions_enable_only.yml
│       ├── mediawiki_extensions_from_repo.yml
│       ├── skins
│       └── ...
└── mysql_data
    └── ...
├── ports.conf
└── switch-canasta-version.sh

Upgrader edit

See https://github.com/dataspects/mediawiki-manager

BlueSpice edit

BlueSpice is a MediaWiki distribution including >100 extensions.

Packaging edit

For packaging, we use composer. Extensions must be registered with a composer registry. Many MediaWiki extensions already are registered in the public composer registry, but it is also possible to maintain a separate registration.

For package definition, we use a set of composer files which are read by MediaWikis composer merge tool.

Example:

This includes the ldap stack.

{
  "repositories": [{
    "type": "composer",
    "url": "https://packages.bluespice.com/"
  }],
  "require": {
    "mediawiki/auth-remoteuser": "2.*",
    "mediawiki/ldap-authentication-2": "1.*",
    "mediawiki/ldap-authorization": "1.*",
    "mediawiki/ldap-groups": "1.*",
    "mediawiki/ldap-provider": "1.*",
    "mediawiki/ldap-user-info": "1.*" 
  }
}

(Source: https://github.com/hallowelt/mediawiki/blob/3.1.x/_bluespice/build/bluespice-ldap/composer.json)

The definitions are currently maintained in a forked repository of MediaWiki, which fetches the MediaWiki changes on a regular basis and adds the composer files into a separate folder.

The repository can be found here (current production branch): https://github.com/hallowelt/mediawiki/tree/3.1.x

Build edit

In order to build, we clone the repository, execute composer install and then package the folder into a tarball.

Deploy edit

The tarball is the basis for a Docker image, which can be found here and is fully functional out of the box:

Docker image definition: https://github.com/hallowelt/docker-bluespice-free
Docker image in repository: https://hub.docker.com/r/bluespice/bluespice-free

Infrastructure Approach Capabilities Comparison edit

			Approach
#	Capability	Is Needed for Canasta? (Y/N)	BlueSpice	dataspects MW Manager	MEZA	Other
OS	Installs	tbd	tbd	by Docker image	tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
Apache	Installs	tbd	tbd		tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
Database	Installs	tbd	tbd		tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
PHP	Installs	tbd	tbd		tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
Caching	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd
ElasticSearch	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd
Profiling	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd
Parsoid	Installs	tbd	tbd	no backward compatibility with MW<1.35	tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
NodeJS	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd
Pywikibot	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd
tbd	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd
Imagemagick	Installs	tbd	tbd	by Docker image	tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
mail	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd
composer	Installs	tbd	tbd	composer.phar	tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
git	Installs	tbd	tbd	no	tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
Mediawiki Core	Installs	tbd	tbd	in the archive	tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
Mediawiki Extensions	Installs	tbd	tbd		tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
Mediawiki Skins	Installs	tbd	tbd		tbd	tbd
	Updates	tbd	tbd		tbd	tbd
	Configures	tbd	tbd		tbd	tbd
Synchs non-infrastructure (wiki pages)	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd
?	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd
?	Installs	tbd	tbd	tbd	tbd	tbd
	Updates	tbd	tbd	tbd	tbd	tbd
	Configures	tbd	tbd	tbd	tbd	tbd

Glossary edit

Term	Definition	Comments
Infrastructure	Anything that is not wiki-text in the wiki database or files in the wiki images folder Rich The "infrastructure" or software stack is the software subsystems (OS, Webserver, database, PHP, etc.) that are required to run the applications (MediaWiki, ElasticSearch, etc.) Bryan
Wiki Content
User Content
Canasta Content
Canasta Packages
Canasta Products
Canasta Tools
MECE	Mutually Exclusive Collectively Exhaustive (MECE) is a grouping principle for separating a set of items into subsets Positive example: categorizing people by year of birth (assuming all years are known). Negative example: categorization by nationality, because nationalities are neither mutually exclusive (some people have dual nationality) nor collectively exhaustive (some people have none).
Idempotent	f(f(x))=f(x) In imperative programming, a subroutine with side effects is idempotent if the system state remains the same after one or several calls, in other words if the function from the system state space to itself associated to the subroutine is idempotent in the mathematical sense given in the definition; In functional programming, a pure function is idempotent if it is idempotent in the mathematical sense given in the definition.