Wikimedia Technical Conference/2018/Session notes/Choosing installation methods and environments for 3rd party users

Theme Defining our products, users and use cases
Type Evaluating Use Cases
Session Leader Daren Welsh
Facilitator Leszek Manicki
Scribes TheDJ, Nick
Phabricator https://phabricator.wikimedia.org/T206059

Description: A key impediment in enterprise use of wikis is the difficulty to install, configure, maintain, and upgrade MediaWiki and all of the extensions required for a modern enterprise knowledge management platform in all but the most basic of environments. There are several efforts underway to address this issue, including meza and efforts within WMF to support containerized installation. This session will seek to identify issues, opportunities, and potential improvements with respect to MediaWiki installation.

Attendees: Daren Welsh, Vogel, FlorianSW, AdamW, Greg G, Adam Basso, Cindy C,  Karsten, Moritz, Leszek, Daniel K, Corey Floyd, mark, Alexandros, Timo, Alexia, Gergo, Mark Bergsma

Questions suggested for the session edit

Question Significance:

Why is this question important? What is blocked by it remaining unanswered?

What level of support is required to support 3rd party installation and configuration? Consider installation of MediaWiki, extensions, and ancillary software as well as preparing the OS for MediaWiki and potentially configuring wiki farm capability.

The following questions assume that at least some support will be provided.

This is a fundamental question in resource allocation. It also guides how WMF staff respond to 3rd party questions and requests on Phabricator, mailing lists, and other communication venues.

There is a spectrum from no support (3rd parties must contribute and support all relevant features), partial support (WMF will answer questions and provide guidance and code review only), to full support (dedicated engineers and product management at WMF for 3rd party features).

How do 3rd parties install MediaWiki currently? What impediments do those approaches to installation present? How critical are those impediments to the use of MediaWiki by 3rd parties? Is it likely that some 3rd parties are not using MediaWiki since they cannot install it as it currently exists? This affects the timeline for providing support. If this problem is considered a low priority by the WMF, it may not be included in the Platform Evolution program. This may constrain the solution set if some solutions require architectural support.
What use cases and methods of installation and configuration should WMF support for 3rd parties? Consider enterprise needs for development, testing, staging, backup, and production environments. Consider wiki farms. Each wiki may be configured differently. This, again, affects resourcing. Is the current tarball approach sufficient, or should a more full-featured approach be developed? If the latter, should it be developed by WMF or 3rd parties? If the latter, how much help will the WMF provide? The choice of method may also put constraints upon supported target environments (e.g. is managed hosting supported? Windows? Alternative databases?)
Should the WMF provide wiki hosting options (paid for and/or free)? If WMF were to provide managed hosting, some constraints upon installation methods may be eased.

Would this help solve the issue of wiki installs on shared hosting providers (because this would allow for VE, Elasticsearch, and other advanced services not provided by shared hosting providers)?

Should we support the packaging of extension backed services for 3rd party installation? Should multiple “editions” be supported? If so, how? If multiple “editions” with different sets of bundled extensions are made available, additional resourcing will be required.
Is it worth the WMF's time and resources to test and develop a production quality container-based approach for 3rd party use, or at least support 3rd party developers of a container-based system?

What, if any, constraints are put on target environments by a container-based approach? Would containers provide a benefit or impediment for getting IT department approval for installation in an enterprise/government environment? Will containers support production as well as development environments? How would a containerized approach scale? What skill level would be required to manage a containerized environment?

Containers have been suggested as a solution to the 3rd party installation problem. Without a decision on whether spending resources to develop and test this approach and without experience-based data, it is difficult to move the discussion forward.
Should an administrator be able to configure an instance through a web based interface? What types of actions should the admin be able to perform through the UI? This is a frequently requested feature by 3rd party users. It should be decided if such a capability can be provided securely and, if so, what architectural support is necessary in core. There may be use cases for community administrators and staff as well (maybe non-technical staff).

Questions actually discussed edit

The following questions were posted on a white board to solicit feedback before the session:

Of the different ways for 3rd party users to install MW and its extension-backed services, what issues/impediments does each method present? Put a dot/star next to the issues you think should be of high priority for the foundation to address.

  • Tarball
  • Git
  • Software bundles (Vagrant, Blue Spice, Meza, etc)
  • Pre-built containers
  • 1-click hosting services
  • Other?

Do you know of any cases where someone can’t install MW and if so, why not?

Please provide reasons for or against the foundation developing, maintaining, and supporting a tool for installation, configuration, and maintenance/updates of a wiki server (including extensions and extension-backed services).

  • Pro
  • Con

Please provide reasons for or against the foundation providing wiki hosting (paid-for and/or free).

  • Pro
  • Con

Here is a photo of the feedback collected:

 
Feedback collected before the session

Concise output from the session edit

Goals edit

  1. Have a common wiki farm management tool (Core platform team is responsible)
    1. “Runtime management”: Automatically pull the right configuration for different wikis on the same installation/runtime. Separating everything that is global. Family name, instances etc.
    2. "Wiki instance managment": Creation and renaming of wikis, extensions, configuration, deleting, etc.
  2. MW platform server instance should have a split off admin panel/tools web GUI
  3. MW platform server instance should have a health check and verification of server status and config interface

Decisions edit

  1. If we commit to an easy-to-use tool for MW platform installation, configuration, and maintenance, then we can drop support of "one-click installs" on shared hosting environments
  2. A special interest group is necessary to further these goals and facilitate implementation

Actions edit

  1. Define use cases for wiki farm management tool and see if existing tools have overlap (e.g. MW Script, Meza, Hydra)
  2. Determine what blockers exist for WMF wiki hosting (free and/or paid-for)
  3. Determine WMF use cases for wiki farm management tool
  4. Create special interest group and form operational concept for how it works with the WMF

Detailed notes edit

  • Intro from Daren, works at NASA, slowly installing extensions, working on upgrades and configurations, started on Windows server without certain access to settings and had to ask admins - sometimes without knowing how they'd fixed it. Now running several servers on CentOS. Project called Meza installs php, apache, Maria DB, and mediawiki and all extensions you might want, helps us upgrade them, auto backups, performance data. - TLDR we've learned a lot the hard way!
  • Started
  • Methods, tarball, vagrant, meza
  • There has been discussion of containers, maybe out of scope ?
  • Do you have full server access or limited access or the most basic ?

Questions: Give us feedback on

  • whether WMF should provide a tool for installation and config and maintenance of the wiki server
  • A lot of points For, and not too much Against this in the feedback. Seems to have reasonable support on the board.
  • Should we provide a single tool vs. should we provide actual hosting.
  • GregG: WMF would not dog-food it (against?), WMF only point, which is really skewed. If for 3rd parties specifically, as a goal (audience), then there would be an argument for it.
  • DanielK: Should it be prioritized, or do we just want it as engineers
  • Gergo: if we expect WMF to resource this, should provide case
  • Florian: Is it definitely a good idea? Should it install on any/all platforms?
  • Florian: how much should it do ? And will that benefit enough people ?
  • Karsten: Is the goal not to make it accessible to a lot of new people
  • Timo: What would be the use-case expected,
  • E.g. mwscript, wmf specific multiversion mess that we use, but a lot of wikifarms could make use of it.  Could maintain collaboratively.
  • Wouldn't use to install a new wiki, but other folks could extend it to do so. Implementation doesn't have to be singular.
  • Timo: I do think we have the same use-cases
  • DanielK: Setup for wikifarm, few hundred wikis, from same config on sameservers - making that reusable could be interesting. But that's very different from first-time install.
  • Lezsek: Looking at existing 3rd party uses?[?]
  • Action item? See what WMF/Gamepedia/Blueprint/Mesa etc have for multifarm wikis, and at least see if there is something we can make re-useable.
  • Florian: Dell? Has a similar set of scripts for this.
  • Cindy: I;ve also got my stuff integrated into Meza, not just NASA.
  • Daren: We (3 existing maintainers) don't want to continue maintaining sole ownership on this (Meza), in case of bad management decisions. Make sure it can get and remain out there for everyone.
  • Adshore: people should not need to install MediaWiki at all, just go to some wiki provider and request a wiki with all the services that surround it.  That's why I'm in favor of bottom right [WMF to offer wiki hosting]. Not sure if the WMF is best org to lead it though.
  • Karsten: managing multiple wikis. Lots of individuals that want to deploy their own fairytale wiki.
  • Moritz: That might be more common than big wikifarms [?] where you need an admin to do setup of an account
  • Cindy: Not everyone can run a hosted env, installation is only the first part, and maybe the smallest.  Upgrades and maintenance are potentially much bigger.
  • Gergo: WMF offering hosting is controversial. Is it still controversial to say it should be a strategic goal for the WMF to ensure that free mediawiki hosting exits (whether or not provided by the WMF)?
  • ACTION: Ask the Board and WMF ED what they think the blockers are on providing something like this.
  • Creating a terms of service, that determines that only people that allow people that work with the system/strategy etc to pay into it.
  • Timo: It does connect with our mission and strategy. Jimmy Wales used to say, WP is the encyclopedia, and Wikia is the rest of the library. How can we prepare knowledge for integration into WP?  We're currently lacking a way for people to write down their content in the first place. The current standard is a page of text, but we can do better than that. If a wikifarm can split subjects[?]
  • ^^^ Adam agrees /Cindy as well
  • Moritz: WMF shouldn't dive into the bits and bolts. Should be more realistic. E.g. blazegraph deploys in different solutions, people say [... is a joke], but then Neptune started and people took it seriously.  Incubator is a good product, and goes further than many out there. But as soon as you talk about a technically unsupported product people don't take it seriously. Building Trust.
  • Overall goal: Make it easier to install and maintain MW.
  • MarkB: giving a repository or platform for Free knowledge and providing hosting for  arbitrary usage is not the same thing.
  • Even if WMF provided shared hosting, it wouldn't meet all these use cases.
  • Gergo: important to make things as easy as possible to get as much traction as possible. Then we would have much more companies providing wiki as a service. Method of getting to the bottom right segment (free wiki hosting exists) is through the top right segment (provide deploy tools etc)
  • Gergo: Want to push back that WMF wouldn't use features. We could use many of these benefits for creating new sister projects. Ties into the topics from the fireside chat.
  • Action item: Exploring ways the WMF could make use of the tool. Using a wikifarm installation tool as a means of addressing the easy creation & maintenance of sister project wikis/experiments.
  • Adam, the foundation could build exactly what is need to provide the free hosting (for earlier reasons), without actually exposing it themselves.
  • Cindy: [?] how enterprise wikis are used.
  • MarkB: WMF is currently working on alternative infrastructure - pipeline - will be moving to k8s. Next year we hope to migrate MediaWiki.
  • That will be very different way of how you would maintain a wikifarm, so take that into account
  • Cindy: people on the outside are using very different architectures that need to be taken into account, and we can learn from
  • Timo: +1 there's a lot of mutual benefit potential. WMF has [??] larger wikifarm than most other existing uses, but our farm isn't inherently more complex, so we shouldn't be using unique tools. All farms need tools to manage them.
  • Gergo: Wikia has 400,000 wikis
  • Gergo: many other things that are broken. Management would see/solve? Different gaps
  • Leszek: How does the goal of common wikigarm management. Is it a principle?[?]
  • Greg: I think for us, yeah would be cool for us and 3rd party users, but not for enterprise/wikia etc. Wikia is so far diverged from us, that it is hard to reconcile on both sides. If we focus on the known ‘smaller’ quataties, it might be worth it.
  • Lezsek: Is it worth it at all to call it a goal. Daniel: Yes.
  • Florian: would solve things on both sides.
  • Timo: re: Should we use the same wiki management goal: actually 2 subgoals - runtime management. The other is "management" from the outside. Creation and maintaining of wikis.
  • Gergo: 1st is a dependency of 2nd. 1st is something that goes deep into wiki core. 2nd is also secondary goal for that reasons.
  • Cindy: Runtime management to me means: is it up, that is not what you mean right ?
  • Timo: I mean i want to create a new wiki, doesn't change code in any way  in WMF case we do change site configuration, but generally not a requirement). Runtime management - how do you load configuration for a wiki?
  • For wmf that is the siteconfiguration standard, but for other platforms, that might be different. Doesn't need to be a commandline tool but..
  • Gergo: Lack of runtime management is a problem for WMF production cluster. Loading data from another wiki, like shared file description page, is hard.
  • Timo: these are problems we are actually facing today. We use SiteConfiguration to get the config for another wiki, but we have lots of variables that are set outside of that method, so often it fails.
  • Cindy: Too many degrees of freedom in how someone sets up a wikifarm.
  • ACTION/DECISION:  
  • Daniel: Wikifarm management is a component WMF should dev for not only their own use. Having wikisites, other sikis, as first class citizens is essential for [crosswiki access?].  The process of creating, identifying, accessing another wiki is essential.
  • Alexia: hydra, we tried using SiteConfiguration, we gave up on that because we had problems and then we made our own thing. Was not scalable to us. We needed info about a wiki, but only had API access.
  • Timo: Is maybe the same problem, we are also not always using it, why WMF bypasses it sometimes as well.  
  • DECISION?: Should be possible for mediawiki to logically instantiate services for a different config, as another wiki within the same farm, without having to go over HTTP.
  • DanielK: So no more globals!?
  • Left-top….
  • Daren: 2 themes between bundle section and 1click - I meant that to be ppl who don't have sudo on their server. I think we agree that everyone needs a better way to install and maintain a wiki, vs the tarball. But do we even need to consider the 1click idea if we make it easy enough to install LAMP plus extension?
  • TheDJ: no prob with individual methods, but always hard to verify that everything i did was correct. No overview of status that shows "everything is ok". Hard to see if everything you're trying to fit together is done well, or where it is broken.  Need more consistency - did my composer work, are my extensions uptodate, does memcache work, is redis available, is there a job-runner available taking stuff off queue. All these things coming together, even as a developer, is hard.
  • Daren: Possible Goal: Having not just improved install/maintenance system, but also a status/overview/system-health/sanity check on your runtime setup.
  • Greg: Basically MediaWiki needs an admin frontend.
  • Cindy: Maybe not in mediawiki, because it needs to work, even if mediawiki breaks
  • Alexia: we have a special page for, but when our system crashes we can't find the thing that made the crash
  • TheDJ: Doesn't even need to be a web tool
  • AdamS: We’ve been saying it for 5 years, when are we gonna do it !
  • Daniel: Makes sense to split the selftest from admin panel. Admin panel raises many questions - security concerns, running updates from webinterface, etc.  But 2 things can be done safely - check software is uptodate, and run a self-test - A hook that all extensions should implement.
  • Having something that can change configuration  that can change from an interface, even more concerns when it can also update the software. They are separate but related.
  • Daren: In meza we have something for that.
  • Florian: Give feedback about things you don’t obviously see in the frontend
  • Daniel: That's why i think it should be separate.
  • Corey: We have to answer this. Should analyze more to determine all the likely benefits we'd get out of this, which would potentially justify resources.
  • Gergo: You mentioned sudo/commandline support. Currently you can install mediawiki without commandline support. This might soon change. Do people think this is important?
  • AdamB: critical question, perhaps because, whether we can spinup non-php services, instead of just dropping in php code and getting another database. With shared hosting cannot run parsoid.
  • Adam: from wikidata/base. If you don’t have queryservice running, then you can’t do anything with it. From that side, the shared hosting is already out the window
  • Daniel: Key question is who is owner of making the decision on giving that answer.
  • [?]: Is it Victoria, or would she throw back to TechComm?
  • Daniel: Not Techcomm, it's a Long term product strategy question.
  • Greg: we never had the answer, always the question as developers. We just need to make a decision.
  • DECISION: We recommend [...]
  • Karsten: Long time I recommended we support environment that don't have commandline. [?] SMW using a separate table. Some people manage to run MW on shared hosting.
  • Florian: One downside, with commandline you also have to manage the server. That is the main difference. Otherwise cost little different. I wouldn’t support shared hosting.
  • Timo: 2 questions: 1: what is the budget amount required for which we want to support MEdiaWiki. Shared vs managed is now almost equal in cost.
  • 2: Expertise and responsibility. I want to be able to install on apache, without having to worry about a process dying. Ability to get there, but not about what you can afford to do.?????
  • Daren: Installing MW used to only require PHP files, Apache, and a MYSQL database, but now it's not so simple. Arch splitting into different things.  Whole community could focus on installation maintenance tool to bring it back to that level of simplicity - that would negate the need for shared hosting.
  • Timo : it’s also about developer usability and easy of use
  • Cindy: Also about having confidence that it works - i've worked in env's where they cannot be specially trained in maintaining this new stack.
  • Daniel: providing a bundle for 3rd parties to install, without ‘expertise’ of services. This is a blocker. If we can ship this, then we can drop the constraint on shared installs/services. 0 config/services of
  • Gergo: Support [?] as an action item. If WMF commits to support containerized or packaged MW, that's easy to install and maintain, on a virtual server, we can drop shared hosting.
  • DECISION: what gergo said
  • Open question for internal and external opsen: is containerized solution the only thing we'd need to support? Will others run a container provided by us? Or do we need to supply other, like debian package?
  • Adam: creating a simple docker file is almost enough to cover all the cases. It provides the supported list of how to get to x. But I'm scared of what the WMF docker would look like!
  • TheDJ: How do we bring the people in this room back again for answering open questions? For pushing meetings and hosting and etc forward.