Deployment tooling/Notes/New branch deploy

See wikitech:Heterogeneous deployment/Train deploys for live documentation of the Wikimedia deployment process for MediaWiki.

Typical process for deploying a new branch of MediaWiki to the wiki[mp]edia production cluster.

Overview edit

  • Create a new 1.XwmfN branch from current HEAD
  • Copy that new version to all MW servers
  • Change all Wikipedias to use 1.XwmfN-1 branch
  • Change group0 wikis (test/test2/testwikidata/mediawiki) to use 1.XwmfN branch
  • Update documentation

Detailed process edit

  1. Run make-wmf-branch script
    1. Branch all extensions
    2. Branch core
    3. Update core branch submodules to reference extension branch HEADs
    This could be done by a Jenkins job or similar automated process
    Should involve a gate process that chooses a stable branch point rather than whatever the state of each HEAD is at the time the process is run
    Should be tested as a unit by regression tests before going to tin
  2. Run checkoutMediaWiki on tin
    1. Checkout branch in a new directory on tin
    2. Initialize submodules
    3. Create symlinks to new branch from bits and w to static assets in branch
  3. Apply any unmerged security patches from N-1 branch
    Should be automated an incorporated into checkout script
    Should be made to be easier to audit to validate that all patches are present, especially for submodules
  4. Git add and commit symlinks
  5. Clean up branches on deploy server that have been inactive for >31 days
    1. Cleanup bits symlinks to branch
      • /a/common/multiversion/deleteMediaWiki php-1.23wmfX
    2. Create symlinks cleanup patch
      • git rm -r docroot/bits/static-1.23wmfX
      • git rm -r w/static-1.23wmfX
      • NOLOGMSG=1 git commit -m 'Remove 1.23wmfX symlinks'
    3. Revert change on tin
      • NOLOGMSG=1 git reset HEAD^ --hard
    4. Repeat steps above for each branch as needed
  6. Run updateWikiversions
    1. Change wikiversions.json to run all on version N-1
  7. Git add wikiversions.json and git commit
  8. Run updateWikiversions
    1. Change wikiversions.json to run group0 on version N
  9. Git add wikiversions.json and git commit
  10. Git push symlink and wikiverisons patches to gerrit
  11. Git reset HEAD~4 working directory and index on tin
    Should be scripted
    Should not require ssh agent forwarding on tin
  12. Review and approve the symlinks patches in gerrit
  13. Wait for zuul & Jenkins to merge patch to branch
  14. Git fetch --rebase working copy on tin
  15. Delete expired branch checkout in /a/common
    • rm -rf /a/common/php-1.23wmfX
    • Repeat steps above for each branch as needed
  16. Manually edit wikiverisons.json to run testwiki on version N
  17. Run scap
    1. Validate PHP syntax in config directory
    2. Copy staging directory to common-local on tin (for utility scripts)
    3. Update l10n cache for version N-1
      1. Use PHP to read all l10n files and create CDB for each language
      2. Create json copies of CDB contents
      3. Create MD5 checksums of json copies
    4. Build l10n cache for version N
      1. Use PHP to read all l10n files and create CDB for each language
      2. Create json copies of CDB contents
      3. Create MD5 checksums of json copies
    5. Copy state of tin to sync slaves via rsync pull
    6. Copy state of sync slaves to all MW servers via rsync pull
    7. Build l10n cache on all WM servers
    8. Compile wikiverisons.json to cdb
    9. Copy wikiverions.{json,cdb} to all MW servers
  18. Revert local changes to wikiverisons on tin
  19. Test testwiki & watch for cluster errors
  20. Review and approve the all wikis on version N-1 patch
  21. Wait for zuul & Jenkins to merge patch to branch
  22. Git fetch --rebase working copy on tin
  23. Run sync-wikiverisons
    1. Compile wikiverisons.json to cdb
    2. Copy wikiverions.{json,cdb} to all MW servers
  24. Test wikis & watch for cluster errors
  25. Review and approve the group0 on version N patch
  26. Wait for zuul & Jenkins to merge patch to branch
  27. Git fetch --rebase working copy on tin
  28. Run sync-wikiverisons
    1. Compile wikiverisons.json to cdb
    2. Copy wikiverions.{json,cdb} to all MW servers
  29. Test group0 & watch for cluster errors
  30. Create/update deploy notes on mw.o
    1. git checkout --track -b wmf/N origin/wmf/N
    2. git submodule update --init --recursive
    3. php uploadChangelog.php wmf/N
    4. git checkout --track -b wmf/N-1 origin/wmf/N-1
    5. git submodule update --init --recursive
    6. php uploadChangelog.php wmf/N-1
    7. git checkout --track -b wmf/N-2 origin/wmf/N-2
    8. git submodule update --init --recursive
    9. php uploadChangelog.php wmf/N-2
    This should be automated to run every day or after every branch merge
    Should use a bot account for upload
    Upload script should use oauth for authentication
  31. Update Roadmap with deploy status