Requests for comment/Extension manager



This page is for getting feedback about implementing an extension management system — reinvigorated by a discussion at the Ask the Developers session at Wikimania 2012. This issue was revisited in the 2013-12-13 tech talk, and probably other places.

Request for comment (RFC)
Extension manager
Component General
Creation date
Author(s) DanielRenfro
Document status declined

Problem(s) edit

  1. There is no explicit dependency and/or compatibility information in the current implementation of extensions.
    This means there is no way to tell if version X of an extension works Mediawiki version Y or not.
    (Except those extensions using Composer, more about that later...)
  2. There is no metadata about an extension available outside of PHP/MW.
    For example, you can't ask who the author is or what version it is.
    (See Requests for comment/Extension registration for a proposal on this topic.)
  3. There is no way to track extension usage, gather feedback, or rate extensions.
    Which wiki's are using Extension:LDAP Authentication? Should the community keep it updated? etc.
  4. There is no way for a sysop to administer extensions (or Mediawiki itself) from within the wiki.
    You currently need CLI access and to know what you're doing (relatively.)
  5. Extensions not in subversion git/gerrit (e.g.: those extensions pasted into the wiki) tend to be not well supported and contain security risks

Solution edit

A secure, integrated, intelligent, automatic, (and slightly distributed) extension manager.

Rationale edit

Mediawiki has a robust system for implementing extensions, but currently lacks an integrated mechanism for managing them. As of the most recent version of Mediawiki (v1.41.1) installation, management, and dependency-resolution of extensions is done manually. These tasks are usually done by a sysadmin or a staff member. While knowledgeable users (with command-line access) can keep the extensions managed, it seems like there should be an integrated solution that does not require ssh.

Benefits edit

  • Provides some tracking and statistics for Mediawiki extensions and updates
  • Wiki admins/sysops can also administer extensions without command-line access/knowledge
  • Faster than doing it manually[1]
  • Should cause people to run more updates (especially if a notification system is built in)[1]
  • More visibility for extensions and extension developers[1]
  • Easier to configure existing extensions — less headaches and manual code-fixes.
  • It's Web2.0-ish![2]

History edit

This issue has come up again and again in various forms and forums (online and offline) over the years. Jeroen De Dauw's Google Summer of Code 2010 proposal, in many extension (see below), Requests for comment/Extension release management started in March 2011, and a few discussions both on and off wiki.[3]

  Note: The lessons learned already should definitely not be ignored — we've got a lot of good coders who've spent lots of time thinking about and solving some of these problems.

Current state of extension management edit

Mediawiki ships with a number of "pre-installed", (or pre-installable) extensions:

  • ConfirmEdit
  • Gadgets
  • Nuke
  • ParserFunctions
  • Renameuser
  • Vector
  • WikiEditor

While the argument of whether or not these extensions should be folded into core is outside the scope of this document, this illustrates a need for a way to package common extensions with the software. A better alternative is to give the configuring user the ability to search and install any and all extensions they feel they need when they are running the install script. Currently this is impossible as the infrastructure doesn't exist – providing a copy with the software is a temporary fix that works well.

Composer edit

TODO: finish this section

Composer


Extensions edit

There have been a number of attempts at solving the problem of extension management. Listed below are extensions directly and indirectly related to this project.

Extension:Distribution, Extension:Deployment edit

Most closely related to this project is Jeroen De Dauw's Google Summer of Code 2010 proposal. Its aim was this exact issue and resulted in the extensions Extension:Distribution and Extension:Deployment. It has influenced this page a lot.[1]

Extension:ExtensionDistributor edit

Extension:Wiki Administration Tool edit

Extension:ExtensionManager edit

This extension is misnamed. Jean-Lou Dupont created Extension:ExtensionManager in order to facilitate installation of other extensions he authored. It seems to be currently unmaintained based on the lack of updates and article history. The idea behind this extension was good—an installer for other extensions, downloaded from a common repository. However, it's scope is too small for the Mediawiki software in general.

Extension:ExtensionClass edit

Extension:ExtensionClass simply adds a PHP class named ExtensionClass, which is used by a number of other extensions used by Jean-Lou Dupont. While possibly useful, it seems hopelessly out of date - similar to the above extension.

Extension:ExtensionInstaller edit

Redundant with the current installer, this extension allows extensions to be bundled and installed with the Mediawiki software. Once setup is complete, this extension doesn't do anything.

Extension:Configure edit

The Configure Extension is quite exciting and powerful extension that allows in-wiki changing of many, many settings. Some of the features include:

  • a history of past configurations so that an admin can easily rollback site-wide configuration to an earlier date
  • api modules allow for programmatic access
  • diffs between past versions of settings
  • multiple special pages for managing core and extensions
  • a PHP extension to make parsing some setting-files faster

This is a large step towards a fully integrated configuration environment that does not require command-line access. I think that this extension's goals somewhat overlap with an extension/package manager and it would be a good idea to collaborate with the authors for a common goal.

Thoughts on client-server model edit

Local repositories edit

At first this project was imagined as in a client-server context, with two separate extensions providing the functionality for each. It was assumed that there would be a centralized repository for extensions, and clients (Mediawik installations) would consume extensions being served from the server. But the question arises: "Who manages the extensions on the server and where to they come from?" If an assumption is made that the repository will be an instance of Mediawiki itself, then some functionality needs to be added to turn it into a repository. Each repository would need another repository of its own to install extensions — this is a recursive problem with no solution. It would defeat the purpose of this project to not allow the repository to manage its own extensions. Therefore, it was decided that the code for this project must cover both client and server functionality, allowing repositories to manage their own extensions. (For example: checkout an extension/package from itself via the file:// or http:// protocol and activate it.)

Question
If all repositories are remote (not local,) where does a repository get its extensions?
Answer
Repositories can be local. Allowing Mediawiki installations to both server and consume extensions. (By default the repository functionality is turned off, but can easily be turned on with a single configuration setting.)

The solution was a simple segregation of the idea of installation with the idea of activation. If all installed extensions are immediately active, repositories cannot host extensions without activating all of them. If, however, installation is separated from activation, any Mediawiki instance can act as both a repository—activating only those extensions that are needed.

Remote respositories edit

Even though some repositories can be local, the assumption is that the vast majority will not. MediaWiki or another WikiMedia site will (be default) be the "one, true repository," but the software should necessarily preclude users from setting up their own.

Repository structure edit

  Note: The repository will not simply server extensions from its extensions/ directory. The infrastructure for the repository will (by necessity,) be more complex than serving up PHP files in directories. Who knows what it will look like, but it will probably involve some database tables (or a separate database,) some compression-handling, and authentication code.

Deliverables edit

Absolute requirements edit

  • Must be deployed on MediaWiki.org
  • Track releases of MediaWiki core - release dates, EOL dates, branch points paths to branches/tags, etc.
  • Track releases of Extensions - same deal
  • Track compatibility of released extensions with core MediaWiki releases
  • Provide this information via special pages and the API
  • Rigorous security and error-checking[4]
  • Don't break the current extensions
    • i.e. The require_once() method of installing extensions will still work.
    • Converting to the new style (a "managed" extension) should be well documented — if not partially scripted.

Features edit

This is a long list, but it can be done!

  • dependency & compatibility resolution
  • configure the extension upon installation, and then again whenever you feel like it
  • both GUI and command-line interface
  • integrated tests for compatibility
  • logging
  • permissions (possibly a new role, group, etc.)
  • flashy Web2.0 interface that makes people go "Ooooo"
  • API modules[1]
  • distributed nature — any package can be installed from any repository
  • automatic and manual checking for extension updates (and updates to MW itself)[1]
  • extension browsing[1]
    • recommendation of similar extensions
    • show a list of most used and most recommended/highest rated extensions
    • keyword search with an option to limit by extension-version/compatibility-with-current-MW-version
  • enable and disable extensions without uninstalling them
  • tracking and analytics infrastructure (possibly by means of the api or just querying the database)

"It'd be cool if..." features edit

  • integration with SVN, git, other version-control systems
  • Autodiscovery information for extensions, so dropped-in extension directories can be automatically listed and selected by such a configuration utility.[5]
  • Fold ExtensionDistributor functionality into this, since we're tracking release information
  • an Extension: namespace (NS_EXT) that's filled with auto-generated pages about each extension, somewhat like MediaWiki's Extension namespace
    • provides a place to view information, urls, versions, rate the extension, leave feedback (?), etc.
    • Make infoboxes on Extension: pages auto-populated with release information, authors, etc.
  • one-click installation[1]
  • a better Special:Version section
  • feedback on extensions in the form of
    1. ratings (1-5 stars?!)
    2. comments
    3. ability to fork w/link to source code (github or tarball or other)
  • downloading class that supports:
    • resuming, timestamp checking, parallel downloads, cached file validation
    • FTP user limit autodetection
    • file://, ftp://, http://, https://, scp://, ftps://, ldap://, etc.
  • easy way to upload a new version of your extension to a repository (api?)
  • special page for configuring mediawiki itself[1][6]
  • parser-functions like {{#latestreleasenumber:extension}}, {{#supportedreleasenumber:extension}}, etc.

Implementation edit

 
One possible installation process for the extension manager.


Prerequisites edit

  1. Chad says "Yes. Let's please please please not try to tackle this until config mgmt is in core." [7]

Hooks edit

  • hooks for the updaters, "for enabled extensions to do their thing"[5]

Database edit

  • some extra database tables, possibly a separate database to hold the extension/packages.

Special pages edit

  • Special:Extensions
    • Provide categorized information about extension(s), release dates, compatibility, path to download, authors, etc.
    • Allow users with appropriate rights to download/upload extensions to/from a repository, activate/deactivate, configure, rate, leave feedback on, and search-for extensions
  • Special:ExtensionRepository
    • An interface for using Mediawiki as a repository for extensions
    • Configuration of connections and protocols (http(s), ftp, ldap, etc.)
    • Provides statistics about down/uploads (of extensions,) users, ratings, feedback

Namespaces edit

  • A defined namespace for displaying pages for each extension — especially documentation!

API edit

  • API modules providing the same information as above.

Command line / maintenance scripts edit

  • Command-line script to allow for all the same functionality for those users who prefer the command-line

References edit

  1. 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 Many of the features and benefits here were not immediately clear to me -- I must give a lot of credit to Jeroen for thinking this through and making thorough notes. I've tried to add a reference to his Google Summer of Code 2010 proposal page whenever it impacted my thinking, directly or indirectly. Thanks Jeroen!
  2. This is not really a benefit, but it would be cool if we had sliding boxes and stuff that fades like all good webapps
  3. Various discussions on-wiki Search for extension management in the following pages (archived mailing list pages):
  4. http://www.cs.arizona.edu/stork/packagemanagersecurity/otherattacks.html
  5. 5.0 5.1 http://lists.wikimedia.org/pipermail/wikitech-l/2007-June/031850.html
  6. Maybe this is getting beyond the scope of the extension/package-manager functionality
  7. http://lists.wikimedia.org/pipermail/wikitech-l/2012-July/062087.html

See also edit