Deployment tooling/Notes/New branch deploy
This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. |
See wikitech:Heterogeneous deployment/Train deploys for live documentation of the Wikimedia deployment process for MediaWiki.
Typical process for deploying a new branch of MediaWiki to the wiki[mp]edia production cluster.
Overview
edit- Create a new 1.XwmfN branch from current HEAD
- Copy that new version to all MW servers
- Change all Wikipedias to use 1.XwmfN-1 branch
- Change group0 wikis (test/test2/testwikidata/mediawiki) to use 1.XwmfN branch
- Update documentation
Detailed process
edit- Run make-wmf-branch script
- Branch all extensions
- Branch core
- Update core branch submodules to reference extension branch HEADs
- This could be done by a Jenkins job or similar automated process
- Should involve a gate process that chooses a stable branch point rather than whatever the state of each HEAD is at the time the process is run
- Should be tested as a unit by regression tests before going to tin
- Run checkoutMediaWiki on tin
- Checkout branch in a new directory on tin
- Initialize submodules
- Create symlinks to new branch from bits and w to static assets in branch
- Apply any unmerged security patches from N-1 branch
- Should be automated an incorporated into checkout script
- Should be made to be easier to audit to validate that all patches are present, especially for submodules
- Git add and commit symlinks
- Clean up branches on deploy server that have been inactive for >31 days
- Cleanup bits symlinks to branch
- /a/common/multiversion/deleteMediaWiki php-1.23wmfX
- Create symlinks cleanup patch
- git rm -r docroot/bits/static-1.23wmfX
- git rm -r w/static-1.23wmfX
- NOLOGMSG=1 git commit -m 'Remove 1.23wmfX symlinks'
- Revert change on tin
- NOLOGMSG=1 git reset HEAD^ --hard
- Repeat steps above for each branch as needed
- Cleanup bits symlinks to branch
- Run updateWikiversions
- Change wikiversions.json to run all on version N-1
- Git add wikiversions.json and git commit
- Run updateWikiversions
- Change wikiversions.json to run group0 on version N
- Git add wikiversions.json and git commit
- Git push symlink and wikiverisons patches to gerrit
- Git reset HEAD~4 working directory and index on tin
- Should be scripted
- Should not require ssh agent forwarding on tin
- Review and approve the symlinks patches in gerrit
- Wait for zuul & Jenkins to merge patch to branch
- Git fetch --rebase working copy on tin
- Delete expired branch checkout in /a/common
- rm -rf /a/common/php-1.23wmfX
- Repeat steps above for each branch as needed
- Manually edit wikiverisons.json to run testwiki on version N
- Run scap
- Validate PHP syntax in config directory
- Copy staging directory to common-local on tin (for utility scripts)
- Update l10n cache for version N-1
- Use PHP to read all l10n files and create CDB for each language
- Create json copies of CDB contents
- Create MD5 checksums of json copies
- Build l10n cache for version N
- Use PHP to read all l10n files and create CDB for each language
- Create json copies of CDB contents
- Create MD5 checksums of json copies
- Copy state of tin to sync slaves via rsync pull
- Copy state of sync slaves to all MW servers via rsync pull
- Build l10n cache on all WM servers
- Compile wikiverisons.json to cdb
- Copy wikiverions.{json,cdb} to all MW servers
- Revert local changes to wikiverisons on tin
- Test testwiki & watch for cluster errors
- Review and approve the all wikis on version N-1 patch
- Wait for zuul & Jenkins to merge patch to branch
- Git fetch --rebase working copy on tin
- Run sync-wikiverisons
- Compile wikiverisons.json to cdb
- Copy wikiverions.{json,cdb} to all MW servers
- Test wikis & watch for cluster errors
- Review and approve the group0 on version N patch
- Wait for zuul & Jenkins to merge patch to branch
- Git fetch --rebase working copy on tin
- Run sync-wikiverisons
- Compile wikiverisons.json to cdb
- Copy wikiverions.{json,cdb} to all MW servers
- Test group0 & watch for cluster errors
- Create/update deploy notes on mw.o
- git checkout --track -b wmf/N origin/wmf/N
- git submodule update --init --recursive
- php uploadChangelog.php wmf/N
- git checkout --track -b wmf/N-1 origin/wmf/N-1
- git submodule update --init --recursive
- php uploadChangelog.php wmf/N-1
- git checkout --track -b wmf/N-2 origin/wmf/N-2
- git submodule update --init --recursive
- php uploadChangelog.php wmf/N-2
- This should be automated to run every day or after every branch merge
- Should use a bot account for upload
- Upload script should use oauth for authentication
- Update Roadmap with deploy status