Library infrastructure for MediaWiki

Group:	Platform Engineering, Wikimedia Engineering Top Priorities FY2014-2015, Q2
Start:	2014-10-20
End:	2014-12-31
Team members:	Kunal Mehta, Aaron Schulz, Chad Horohoe
Lead:	Bryan Davis
Management:	Rob Lanphier (engineering director)

Rationale

Our long-term goal is to make MediaWiki an application that is composed of many small purpose-built libraries (internally and/or externally developed) with interfaces that allow individual libraries to be exchanged for others. This flexibility allows easier tuning of major features for various use cases, and easier development of new large features by composition. For example, a (far) future project might replace the current revision storage with a system that has better performance characteristics for the Wikipedia wiki farm, simply by configuring an alternate implementation of the revision storage library rather than including two implementations and branching logic in the MediaWiki codebase.

Make life better for new (and experienced) developers by organizing the code into simple components that can be easily understood.
Reverse inertia toward ever expanding monolithic core by encouraging developers (in core) to develop their work as reusable modules with clearly-defined interfaces
Start making true unit testing of core viable by having individually-testable units
Provide an interim step on the way to service-oriented architecture in a way that is useful independently of that goal
Encourage reuse and integration with larger software ecosystem. Done correctly, this will provide a useful means of expanding our development capacity through recruitment of library authors eager to showcase their work on a top 10 website.
Share our awesome libraries with others and encourage contributions from them even if they aren't particularly interested in making our sites better.

Out of scope

Complete breakup of MediaWiki core into components.

The end goal is not to eliminate MediaWiki but instead to make MediaWiki the platform for creating massive multi-user versioned data-based projects, and make it a platform that is flexible enough to extend in novel ways. (For internal refactoring of classes and namespaces in the MediaWiki codebase to break dependency cycles, see phab:tag/mediawiki-decoupling instead.)

Timeline

FY2014-15, Q2

Complete implementation and deployment of Structured logging RFC
- Status: Done Introduce Composer library management to MediaWiki's core git repository
- Status: Done Introduce mediawiki/vendor git repository to manage libraries for WMF cluster deployments
- Status: Done Introduce PSR-3 logging interface to MediaWiki
- Status: Done Introduce optional external library (Monolog) to implement PSR-3 interface

Complete work for Composer managed libraries for use on WMF cluster RFC
- Document use of Composer for library management
- Procedures officially approved by architecture, security and operations groups
- Status: In progress Implement monitoring tools to track security vulnerabilities in external libraries

Extract one component from MediaWiki, publish as a stand alone library and re-import to MediaWiki via Composer
- Status: Done Establish guidelines for git hosting, issue tracking and community management for independent libraries
- Status: Done cssjanus
- Status: Done cdb

Status: Done Create list of future candidates for library extraction projects

Status: WONTFIX Investigate implementing a library management system for JavaScript libraries used by MediaWiki
- task T807
- Discussions with the Frontend standards group concluded that at this time the future of ResourceLoader and the general javascript package management space are too uncertain to commit to implementing automated javascript package management for MediaWiki. They would however like to participate in discussions of standards and practices for creating or extracting libraries and ensuring they are properly managed as independent or at least semi-autonomous open source projects. The group would like to revisit this topic in 3-6 months when currently active discussions about the future feature set of npm have been resolved. See meeting minutes for additional details.

Documents

Phabricator librarization project and its workboard
Manual:External libraries
Manual:Developing libraries