Requests for comment/Extension manager

See also: Requests for comment/Extension management with Composer and Requests for comment/Extension release management

This page is for getting feedback about implementing an extension management system — reinvigorated by a discussion at the Ask the Developers session at Wikimania 2012. This issue was revisited in the 2013-12-13 tech talk, and probably other places.

Request for comment (RFC)

Extension manager
Component	General
Creation date	29 July 2012
Author(s)	DanielRenfro
Document status	declined

Problem(s)

1. There is no explicit dependency and/or compatibility information in the current implementation of extensions.

   This means there is no way to tell if version X of an extension works Mediawiki version Y or not.

   (Except those extensions using Composer, more about that later...)

2. There is no metadata about an extension available outside of PHP/MW.

   For example, you can't ask who the author is or what version it is.

   (See Requests for comment/Extension registration for a proposal on this topic.)

3. There is no way to track extension usage, gather feedback, or rate extensions.

   Which wiki's are using Extension:LDAP Authentication? Should the community keep it updated? etc.

4. There is no way for a sysop to administer extensions (or Mediawiki itself) from within the wiki.

   You currently need CLI access and to know what you're doing (relatively.)

5. Extensions not in subversion git/gerrit (e.g.: those extensions pasted into the wiki) tend to be not well supported and contain security risks

Solution

A secure, integrated, intelligent, automatic, (and slightly distributed) extension manager.

Rationale

Mediawiki has a robust system for implementing extensions, but currently lacks an integrated mechanism for managing them. As of the most recent version of Mediawiki (v1.42.3) installation, management, and dependency-resolution of extensions is done manually. These tasks are usually done by a sysadmin or a staff member. While knowledgeable users (with command-line access) can keep the extensions managed, it seems like there should be an integrated solution that does not require ssh.

Benefits

• Provides some tracking and statistics for Mediawiki extensions and updates
• Wiki admins/sysops can also administer extensions without command-line access/knowledge
• Faster than doing it manually
• Should cause people to run more updates (especially if a notification system is built in)
• More visibility for extensions and extension developers
• Easier to configure existing extensions — less headaches and manual code-fixes.
• It's Web2.0-ish!

History

This issue has come up again and again in various forms and forums (online and offline) over the years. Jeroen De Dauw's Google Summer of Code 2010 proposal, in many extension (see below), Requests for comment/Extension release management started in March 2011, and a few discussions both on and off wiki.

 Note: The lessons learned already should definitely not be ignored — we've got a lot of good coders who've spent lots of time thinking about and solving some of these problems.

Current state of extension management

Mediawiki ships with a number of "pre-installed", (or pre-installable) extensions:

• ConfirmEdit
• Gadgets
• Nuke
• ParserFunctions
• Renameuser
• Vector
• WikiEditor

While the argument of whether or not these extensions should be folded into core is outside the scope of this document, this illustrates a need for a way to package common extensions with the software. A better alternative is to give the configuring user the ability to search and install any and all extensions they feel they need when they are running the install script. Currently this is impossible as the infrastructure doesn't exist – providing a copy with the software is a temporary fix that works well.

Composer

TODO: finish this section

Composer

Extensions

There have been a number of attempts at solving the problem of extension management. Listed below are extensions directly and indirectly related to this project.

Extension:Distribution, Extension:Deployment

Most closely related to this project is Jeroen De Dauw's Google Summer of Code 2010 proposal. Its aim was this exact issue and resulted in the extensions Extension:Distribution and Extension:Deployment. It has influenced this page a lot.

Extension:ExtensionManager

This extension is misnamed. Jean-Lou Dupont created Extension:ExtensionManager in order to facilitate installation of other extensions he authored. It seems to be currently unmaintained based on the lack of updates and article history. The idea behind this extension was good—an installer for other extensions, downloaded from a common repository. However, it's scope is too small for the Mediawiki software in general.

Extension:ExtensionClass

Extension:ExtensionClass simply adds a PHP class named ExtensionClass, which is used by a number of other extensions used by Jean-Lou Dupont. While possibly useful, it seems hopelessly out of date - similar to the above extension.

Extension:ExtensionInstaller

Redundant with the current installer, this extension allows extensions to be bundled and installed with the Mediawiki software. Once setup is complete, this extension doesn't do anything.

Extension:Configure

The Configure Extension is quite exciting and powerful extension that allows in-wiki changing of many, many settings. Some of the features include:

• a history of past configurations so that an admin can easily rollback site-wide configuration to an earlier date
• api modules allow for programmatic access
• diffs between past versions of settings
• multiple special pages for managing core and extensions
• a PHP extension to make parsing some setting-files faster

This is a large step towards a fully integrated configuration environment that does not require command-line access. I think that this extension's goals somewhat overlap with an extension/package manager and it would be a good idea to collaborate with the authors for a common goal.

Local repositories

At first this project was imagined as in a client-server context, with two separate extensions providing the functionality for each. It was assumed that there would be a centralized repository for extensions, and clients (Mediawik installations) would consume extensions being served from the server. But the question arises: "Who manages the extensions on the server and where to they come from?" If an assumption is made that the repository will be an instance of Mediawiki itself, then some functionality needs to be added to turn it into a repository. Each repository would need another repository of its own to install extensions — this is a recursive problem with no solution. It would defeat the purpose of this project to not allow the repository to manage its own extensions. Therefore, it was decided that the code for this project must cover both client and server functionality, allowing repositories to manage their own extensions. (For example: checkout an extension/package from itself via the file:// or http:// protocol and activate it.)

Question

If all repositories are remote (not local,) where does a repository get its extensions?

Answer

Repositories can be local. Allowing Mediawiki installations to both server and consume extensions. (By default the repository functionality is turned off, but can easily be turned on with a single configuration setting.)

The solution was a simple segregation of the idea of installation with the idea of activation. If all installed extensions are immediately active, repositories cannot host extensions without activating all of them. If, however, installation is separated from activation, any Mediawiki instance can act as both a repository—activating only those extensions that are needed.

Remote respositories

Even though some repositories can be local, the assumption is that the vast majority will not. MediaWiki or another WikiMedia site will (be default) be the "one, true repository," but the software should necessarily preclude users from setting up their own.

Repository structure

 Note: The repository will not simply server extensions from its extensions/ directory. The infrastructure for the repository will (by necessity,) be more complex than serving up PHP files in directories. Who knows what it will look like, but it will probably involve some database tables (or a separate database,) some compression-handling, and authentication code.

Deliverables

Absolute requirements

• Must be deployed on MediaWiki.org
• Track releases of MediaWiki core - release dates, EOL dates, branch points paths to branches/tags, etc.
• Track releases of Extensions - same deal
• Track compatibility of released extensions with core MediaWiki releases
• Provide this information via special pages and the API
• Rigorous security and error-checking
• Don't break the current extensions
  • i.e. The require_once() method of installing extensions will still work.
  • Converting to the new style (a "managed" extension) should be well documented — if not partially scripted.

Features

This is a long list, but it can be done!

• dependency & compatibility resolution
• configure the extension upon installation, and then again whenever you feel like it
• both GUI and command-line interface
• integrated tests for compatibility
• logging
• permissions (possibly a new role, group, etc.)
• flashy Web2.0 interface that makes people go "Ooooo"
• API modules
• distributed nature — any package can be installed from any repository
• automatic and manual checking for extension updates (and updates to MW itself)
• extension browsing
  • recommendation of similar extensions
  • show a list of most used and most recommended/highest rated extensions
  • keyword search with an option to limit by extension-version/compatibility-with-current-MW-version
• enable and disable extensions without uninstalling them
• tracking and analytics infrastructure (possibly by means of the api or just querying the database)

"It'd be cool if..." features

• integration with SVN, git, other version-control systems
• Autodiscovery information for extensions, so dropped-in extension directories can be automatically listed and selected by such a configuration utility.
• Fold ExtensionDistributor functionality into this, since we're tracking release information
• an Extension: namespace (NS_EXT) that's filled with auto-generated pages about each extension, somewhat like MediaWiki's Extension namespace
  • provides a place to view information, urls, versions, rate the extension, leave feedback (?), etc.
  • Make infoboxes on Extension: pages auto-populated with release information, authors, etc.
• one-click installation
• a better Special:Version section
• feedback on extensions in the form of
  1. ratings (1-5 stars?!)
  2. comments
  3. ability to fork w/link to source code (github or tarball or other)
• downloading class that supports:
  • resuming, timestamp checking, parallel downloads, cached file validation
  • FTP user limit autodetection
  • file://, ftp://, http://, https://, scp://, ftps://, ldap://, etc.
• easy way to upload a new version of your extension to a repository (api?)
• special page for configuring mediawiki itself
• parser-functions like {{#latestreleasenumber:extension}}, {{#supportedreleasenumber:extension}}, etc.

Implementation

One possible installation process for the extension manager.

Prerequisites

1. Chad says "Yes. Let's please please please not try to tackle this until config mgmt is in core."

Hooks

• hooks for the updaters, "for enabled extensions to do their thing"

Database

• some extra database tables, possibly a separate database to hold the extension/packages.

Special pages

• Special:Extensions
  • Provide categorized information about extension(s), release dates, compatibility, path to download, authors, etc.
  • Allow users with appropriate rights to download/upload extensions to/from a repository, activate/deactivate, configure, rate, leave feedback on, and search-for extensions

• Special:ExtensionRepository
  • An interface for using Mediawiki as a repository for extensions
  • Configuration of connections and protocols (http(s), ftp, ldap, etc.)
  • Provides statistics about down/uploads (of extensions,) users, ratings, feedback

Namespaces

• A defined namespace for displaying pages for each extension — especially documentation!

API

• API modules providing the same information as above.

Command line / maintenance scripts

• Command-line script to allow for all the same functionality for those users who prefer the command-line

See also

• Requests for comment/Extension registration — similar RFC for registration of extensions
• Composer — dependency manager
• Requests for comment/Extension release management — handling extension releases, somewhat overlaps with Composer
• User:Jeroen De Dauw/GSoC2010/Proposal — handling extension releases
• Extension:ExtensionDistributor
• Extension:Deployment
• Extension:Distribution
• Extension:Update
• Deployment