Talk:Release Management RFP/2014/Consortium

In-place updates

Forgive the ignorance, I understand the optional schema changes but I'm not sure what the first two bullets imply. (Was SVN better than Git at this, by they way?) So my question is: will you always offer .patch files from one minor (and major?) release to the next, as some people asked on mediawiki-l? --Nemo 09:38, 14 June 2014 (UTC)

That section is referring to the process to update the database (I've changed the header to specify this now). Generally switching branches in git or unpacking a tarball or whatever folks wind up doing on the file end should only take a few seconds, and we don't want to go messing up anyone's workflows there without very good reason. Patches are pretty simple to do as well, so I don't see any reason to change that either.

I can't really go into specifics because this is not even remotely my area of expertise, but basically the issues we've run into seem to be with how the schema changes and whatnot are applied. While they're supposed to be optional, they should also be efficient, but the updater often doesn't do them in efficient ways even when there are better options. There are cases, too, when an update may require the entire database to be made read-only or taken down for the duration of the update, but these are only some of the changes, and it should be possible to handle such changes separately in order to minimise downtime. Indeed, most large projects, including the WMF, simply don't use the included updater at all, writing their own scripts to perform the needed changes, but while this makes sense in their case(s) given how their databases are set up, this shouldn't be a necessity in general for anything over 10GB or whatever random number.

Also something about having a sensible, generic, supported way to update the database before updating the files so it all just works right away. As a sysadmin, I want that. I think others want that. We'd be happier if we had that. But it's kind of weird to actually do, or even necessarily explain without handwaving. -— Isarra ༆ 16:11, 14 June 2014 (UTC)

(Edit conflict) Excellent question. For the first bullet point (not take the site down), there are currently a few parts of the upgrade process that often mean site downtime that we believe can be streamlined or mitigated:

• Backups. For large databases, a backup can take quite a while. One could do a backup while the wiki is still up (and indeed that is probably fine for the majority of circumstances), but you risk data loss in the event that edits or other events happen after the backup completes should you need to rollback. While I don't know of any novel ways to easily remedy that, we can reduce the chance of needing a rollback by implementing a thorough testing framework to ensure that in-place upgrades succeed in a wide variety of scenarios.
• Backfilling data as part of schema changes. The updater script currently goes top-to-bottom on a list of pre-set upgrades. Should any of these upgrades necessitate a schema change that requires data to be backfilled (e.g. moving columns to a new table or something), this backfilling should occur as late as possible so that it does not hold up other schema updates from taking place. As long as it is feasible for the feature to mostly work even if some of the data is missing, these updates could take place as part of the job queue instead of being part of the update process. Otherwise, they should be moved as far back in the update list as feasible. The goal here is that all of the schema updates finish before large data copy or aggregation operations so that the wiki is at least usable while these operations complete rather than throwing errors.

Neither of those points really have anything to do with .patch files, although I could see the benefit of offering either a .patch file or a tarball containing only the modified files from the previous version to the most recent (just going back one version though), this reduces the time it would take to upload changes to the server, which matters most for users on shared hosting services where they do not have SSH and are restricted to FTP and the web upgrader, but would assist many other types of users as well. Hope that answers your questions! --Skizzerz 16:16, 14 June 2014 (UTC)

Oh, sorry for misunderstanding the section completely. :"| (But .patch files are not so obvious, if you offer them update the page.) True, schema changes are a major pain point (imagine for translatewiki.net which runs master...). --Nemo 18:40, 14 June 2014 (UTC)

I filed bugzilla:66809 for this issue as I thought it deserved a bug. Bawolff (talk) 20:00, 18 June 2014 (UTC)

Release branches for extensions

«Release branches should be properly verified to work before being published», WOW. If you managed to, you'd the ultimate heros of MediaWiki hosts. To be more realistic, you may want to draw a line somewhere: not 3000 extensions existing in the wild, but "only" the few hundreds in gerrit; or only the N hundreds which are above X usage outside Wikimedia wikis; or only the ~hundred in use on Wikimedia wikis; whatever. --Nemo 09:44, 14 June 2014 (UTC)

Aye, we can't account for wild extensions (I'm just going to start calling them that) unless they're specifically brought to our attention, and branching extensions for the different releases is similarly something that can only be guaranteed with the ones in gerrit. Generally speaking Wikimedia can take care of itself so those extensions should be pretty quick to check, but there are many other extensions that are commonly used as well. If nobody seems to be using an extension it's not going to take priority, but all users should need to do is ask in such cases. Hopefully with effective outreach to third-party sysadmins and developers (something we think Wikiapiary will be of considerable help with) we'll be able to get an idea of their priorities here, and even convince some developers to release their wild extensions into the main pile, considering the potential benefits - from security reviews, to translations, to release testing and branching. And then other projects could benefit too. -— Isarra ༆ 16:11, 14 June 2014 (UTC)

There is probably some automated things that could be done. Have a test to require_once, the extension, see if there are any fatals. If its a tag extension (which describes 90% of our extensions), there could be an automated invocation of the tag, and then check for exceptions/fatals. I'm not sure the extent that could be automated, and it may still be a lot of work, but its imaginable that we could have entirely automated smoke tests. Bawolff (talk) 21:19, 14 June 2014 (UTC)

Simply checking for fatal errors/warnings would work in some cases, but for widely-deployed extensions unit tests should be written to cover their main functionality, if such tests do not already exist. Having unit tests pass when the extension is included would be a very good way to be mostly sure that an extension is or is not compatible with a specific release. Another method of automatically detecting incompatible extensions is to maintain a list of removed/changed things that could impact extensions, and then grep for those. Any extensions that return results may have incompatibility issues. --Skizzerz 03:02, 15 June 2014 (UTC)

Guides for sysadmins and developers for updates

So you'd handle what's IMHO the single most important documentation deficiency of MediaWiki after help pages, i.e. Thread:Project:Current issues/Installation guide consolidation? :) --Nemo 09:48, 14 June 2014 (UTC)

That's a scary heap, but aye, in a nutshell. While the documentation is there for anyone and everyone to edit, and there are probably quite a few folks who know more about the specifics than we do, it doesn't do anyone much good scattered all over the place. Consolidating the heap and making sure there's at least something there that's useful and gives a functional overview, and making sure folks can find it and that it's actually reasonably up to date, is something we'd very much like to do, because indeed, it really is a major pain point for third parties. Given the contract we believe we'd be in a position to much more feasibly address this considering how much it would overlap with the rest of what we'd be doing. -— Isarra ༆ 16:11, 14 June 2014 (UTC)

Fallback

As I alluded to at Thread:Talk:Release Management RFP/Cost v. benefit, I'm tentatively supportive of the fallback plan. Free software shouldn't cost $75,000/year to release. --MZMcBride (talk) 19:00, 18 June 2014 (UTC)

You're right, it shouldn't. Personally I'm all for the fallback; to get the tarballs and whatnot out, it's really all we need, and I would hope nobody loses sight of this. Essentially free releases have been very effective in the past, and though they have perhaps more recently left something to be desired in terms of reliability, we know they work.

The money isn't so much for the releases themselves, however, as it is for the stuff that wouldn't necessarily happen anyway. The money is for the time commitment for someone to make the releases a priority and make sure that they go out bug-free and that they go out on time. It's for the consideration specifically given to third-party projects, giving what they need a better chance of getting in, without biasing any particular project over any others (they can spend their own money for that). It's for the auxiliary development that wouldn't otherwise happen, or wouldn't otherwise make it upstream. It's for someone to consistently bother to do tedious stuff like mind the documentation, and go through all the release notes, and start relevant RfCs for things most nobody really cares about but could be rather important, and ensure everything gets reported to all the different lists, and track down and possibly replace package maintainers for various distros when the packages get out of date, and help third-party projects better combat spam and stay up to day and perhaps even contribute upstream, and spend time actually hearing out and addressing issues brought up by end users. All of this could happen without money, of course, but would it?

Certainly sometimes, but we'd like it to be better than sometimes. Though we currently do a lot of this in our spare time, we only have a finite amount of time. We have bills to worry about, and things we'd rather be doing, because some of this, despite its importance, isn't particularly fun. So the money is needed, not for the release management itself, but to spare the time and purchase the energy required to make everything else happen. With some amount of certainty and accountability, that is. -— Isarra ༆ 01:54, 19 June 2014 (UTC)