Wikimedia Platform Engineering/MediaWiki Core Team/Check-ins/20140113

Who: Brad, Chad, RobLa, Aaron, Chris, Nik, Bryan, Ori, Antoine

Escalations edit

Need post-mortem for CategoryTree related downtime earlier today. Greg to followup with Sam

Bug 59957: Gerrit host has weird CPU usage Gerrit outage over the weekend. Chad will investigate + update bug.

Search edit

  • lsearchd hasn’t updated in a week (bug 59979)
    • Chad and Nik have looked, somewhat stumped.
    • Going to research further, will prod Ori/Tim/ops as needed
    • We’re poking Leslie on IRC now
  • Cirrus now secondary search backend on enwiki
  • enwiki job queue backed up (not our fault, but we noticed)
    • 1.7mil htmlCacheUpdate jobs and rising (+30k since I started watching it this morning)
    • Raising job runners by 1 per box: https://gerrit.wikimedia.org/r/#/c/107189/
      • Seems to have helped, numbers going down again (-45k, roughly?)

DevOps edit

  • Aaron documented deployment changes in Jan 9 e-mail to engineering@
  • Bryan to sit in with Max tomorrow
  • Aaron looking into new magic word handling and job queue issues

Performance edit

  • Aggregated and plotted page load timing data for last 10 months: <http://noc.wikimedia.org/~ori/metrics/page-load.html>.
  • TODO: Automate the analysis and produce a weekly report to engineering@ / wikitech-l.
  • Presented at January Metrics Meeting:  
  • Lots of UniversalLanguageSelector analysis and back-and-forths prompted by bug 59958. Reviewed architecture in bug 56433#c11, culminating in Erik’s decision in 56433#c22 to disable automatic font loading, which is a good outcome.
  • Tracked down core test failure under HHVM in bug 55532#c2. Good news: fixed upstream.
  • Tim’s xhprof patch merged to HHVM: https://github.com/facebook/hhvm/pull/1488
  • Andrew Otto ready to configure profiling stats endpoint on bits, waiting on me to e-mail ops@ to get a nod of approval from Faidon or Mark.
  • Still thinking about performance guidelines for developers.

Beta / CI edit

  • Extensions on beta no more self deploying bug 49847
    • mediawiki/extensions.git no more updated by Gerrit. Need to write some script to make it happen for us.

Security edit

Scholarships edit

  • ~80 applications received so far.
  • No serious bugs found but several small changes to text and validation have been requested and deployed.

Quarterly review edit

https://www.mediawiki.org/wiki/Wikimedia_MediaWiki_Core_Team/Quarterly_review,_January_2014

Architecture Summit edit

https://www.mediawiki.org/wiki/Architecture_Summit_2014

https://www.mediawiki.org/wiki/Architecture_Summit_2014/Straw_poll Things not yet on the agenda that should be:

  • Ori: Code review standards
  • RobLa; RFC lightning round