Wikimedia Platform Engineering/MediaWiki Core Team/Check-ins/20140127

Who: Bryan, Nik, Chad, RobLa, Ori, Greg, Chris, Sam, Aaron, Dan, Brad

Quarterly review

Review action items from quarterly review:

• Sumana & Ken: follow up on possibility of "has signed an NDA" LDAP group
• Bryan & Chris: look into 2FA or similar for Logstash Authentication for users
  • Apache + 2FA: https://code.google.com/p/google-authenticator-apache-module/
  • Apache + OpenID: http://auth-openid.sourceforge.net/
  • Apache + LDAP: https://httpd.apache.org/docs/2.0/mod/mod_auth_ldap.html (if none of 2FA options work; ie what we are using now)
• Chad & Nik: Get Brandon a link to a JSON API
• [Chad] I forgot about this. JSON for Elasticsearch? Can easily ping re: this.
• [Nik] He wants the output from MediaWiki but Elasticsearch influences MediaWiki so that should be ok.
• More benchmarks for HHVM & MediaWiki - characterise & pinpoint & quantify benefits of HHVM so we have a real value proposition for rest of org
• Mark B to look into this: To collect frontend performance data, would be great to have a varnish kafka topic running on bits varnishes actiing as aggregation point, asks Ori. Not urgent
  • Separate eventlogger load-balancing IP? suggests Faidon
• Look into provisioning baremetal performance testing infra?
  • maybe just an additional job runner for testing HHVM
• Faidon & Gabriel: Look into provisioning hardware for the large users of Labs, eg Parsoid
• Describe what to do in the event of a users/admin settings leak
  • script it?
  • Chad: figure out why we still have AdminSettings lingering around. I killed that years ago.
• Chris S & Sumana: talk about upcoming training, brainstorm approaches

Search

• This week we’ll start huwiki
• Next week we’ll go BetaFeature on enwiki (WOOOOOO!)
• Nik is working on lots of accuracy and index size and performance issues
• lsearchd is updating again

Deploy Tooling

We had the crazy meeting on Wednesday. Greg will be posting the pictures and sending out a couple emails to start discussions on a couple of the red post-its.

Performance

• Impact of switching ULS to opt-in: bytes out on bits varnishes at 1-year low, median page load times down by 200ms - 400ms, 75th percentile page load times down by almost a second.
• RUM graphs moved to Graphite; added stub dashboard at http://gdash.wikimedia.org/dashboards/frontend/
• Configured http://performance.wikimedia.org/, will have additional dashboards.
• WikimediaShopLink DOM reflow fixed: https://bugzilla.wikimedia.org/show_bug.cgi?id=57939

Beta / CI

• Antoine out :(

Gerrit

• Upgrading to 2.8.1 this evening (starting 5:30ish SF time)

Security

• Release tomorrow for bug 60339 (core/pdfhandler). 1.22 bugfixes will be included.

Bug escalation

https://bugzilla.wikimedia.org/show_bug.cgi?id=54193 600-700 times per hour on prod

• Tim thinks it could be fixed by initializing User sooner https://bugzilla.wikimedia.org/show_bug.cgi?id=41201

Architecture Summit

Followups