HHVM/status

Last update on: 2014-11-monthly

2011-05-13

• Discussion at the Berlin Hackathon 2011 (notes)

2011-04-30

Tim Starling implemented basic support for HipHop for PHP in MediaWiki, and invited other developers to improve and continue his work. We will pick this work back up later this year after the completion of some of the other projects above.

2011-06-01

HipHop was discussed during the Berlin hackathon, and it was agreed that HipHop support would be part of the MediaWiki 1.20 release.

2011-07-01

HipHop support is still planned to be part of MediaWiki 1.20. In the meantime, we're looking for volunteers to help us package it for different distributions. Please contact Sumana Harihareswara or the wikitech-l mailing list if you have experience with packaging or would like to get involved in this area.

2011-07-24

This project was mainly on hold in July.

2011-11-23

Mark Hershberger and Tim Starling, with volunteer Mike Dupont, are seeking help with Debian packaging for HipHop.

2011-12-13

Facebook has announced that it will be providing virtual machines for HipHop. That will aid in developing for it. The Roadmap now calls for Platform Engineering and Ops to make more decisions and plans regarding HipHop in February 2012.

2011-12-31

After Facebook announced that they were developing virtual machines for HipHop, Tim Starling indicated that Wikimedia would put their current efforts on HipHop on hold, until the virtual machines can be evaluated. Other performance efforts like Wikitext scripting will take priority instead.

2013-05-monthly

Several engineers at the Wikimedia Foundation met with Facebook engineers to discuss potential deployment of HHVM in 2013 (summary). We formed the HipHop mailing list to discuss moving forward with this work.

2013-06-monthly

A Labs instance of MediaWiki running on HipHop is now available at http://hhvm.wmflabs.org.

2013-07-monthly

HipHop work was mainly on hold in July, except for some minor work on virtual machines.

2014-02-18

Work is starting back up on this, with the goal of having at least one production service running on HipHop by the end of the quarter. Tim Starling is working with the HHVM upstream to finish off a compatibility layer for running Zend extensions (ext_zend_compat) under HipHop, with the goal of using it for our Lua module. Ori Livneh is working on packaging and deployment issues, as well as generally wrangling the overall development effort. Aaron Schulz is starting to investigate what is needed for wmferrors support.

2014-02-monthly

Work is starting back up on this project, with the goal of having at least one production service running on HipHop by the end of the quarter. Tim Starling is working with the HHVM upstream to finish off a compatibility layer for running Zend extensions (ext_zend_compat) under HipHop, with the goal of using it for our Lua module. Ori Livneh is working on packaging and deployment issues, as well as generally wrangling the overall development effort. Aaron Schulz is starting to investigate what is needed for wmferrors support.

2014-03-monthly

The team continued to work on porting C extensions to HHVM. Tim Starling did major work on a compatibility layer allowing Zend extensions to be used by HHVM, and started further work on making the layer compatible with newer HHVM interfaces. The team has made a preliminary deployment of HHVM to the Beta cluster, but this still needs further debugging before it is useful to a wider audience.

2014-04-monthly

Work on the Zend plugin compatibility layer is feature complete, and now the team is working on proper packaging of HHVM, and is working toward making HHVM the default PHP implementation on the Beta cluster.

2014-05-28

Three fronts: HHVM proper:

• Two segfault bugs (third chased down by Tim to upstream bug):
  • HHVM segfaults when calling Parser->callParserFunction https://bugzilla.wikimedia.org/show_bug.cgi?id=65796
  • LuaSandbox segfaults under HHVM

https://bugzilla.wikimedia.org/show_bug.cgi?id=65792

Production environment:

• Cleanup of mediawiki puppetization continues
• Next up: refactor of apache module, Ori to propose on ops list later today
  • Why: apache module is 2.2-specific; could not be made easily compatible with 2.4 without big changes (including moving apache-config/* to operations-puppet so the files could be templatized); when it was brought up before on the ops list everyone seemed to agree the module sucks and needs to die
  • Is 2.4 really so different? -- Part of the issue is that Debian changed the file layout for Apache config files; there’s no longer an /etc/apache2/conf.d/, etc.

Packaging / release management

• Faidon did a ton of packaging work: http://anonscm.debian.org/gitweb/?p=collab-maint/hhvm.git;a=shortlog
• Also in touch with Paul T. and David M. about outstanding issues. Upstream is aware and willing to work with Debian to get a package out, including cherry-picking patches for a 3.1.1 release
• We’re still going to use our own packages
• We’ll most likely continue to use Tim’s dev branch as the target, cherry-picking fixes to issues we encounter, and then target 3.2 once it comes out in July

2014-05-monthly

HHVM is running on a test machine ("osmium") in our production cluster. Most of Tim Starling's work on the Zend compatibility layer have landed in HHVM 3.1. Most jobs are working, but bugfixing continues on osmium.

2014-06-monthly

The team has been running HHVM on a single test machine ("osmium") for the purpose of testing the job queue in production. The machine is only put into production on a very limited basis, while enough bugs are found to keep the team busy for a while, and then it's disabled again as the team fixes those bugs. We're planning on having HHVM running on a few job runner machines (continually) in July, then turning our focus toward running HHVM on the main application servers, taking a similar strategy.

2014-07-monthly

The Beta cluster is running HHVM. The latest MediaWiki-Vagrant and Labs-vagrant use HHVM by default.

2014-08-16

• Since migrating test.wikipedia.org to HHVM exactly one week ago, we've had just one segfault (reported upstream: <https://github.com/facebook/hhvm/issues/3438>). That's very good.
• Giuseppe shared some benchmarks in an e-mail to wikitech: <https://lists.wikimedia.org/pipermail/wikitech-l/2014-August/078034.html>. Also very good.
• Re-imaging an app server was surprisingly painful, in that Giuseppe and I had to perform a number of manual actions to get the server up-and-running. This sequence of steps was poorly automated: update server's salt key -> synchronize Trebuchet deployment targets -> fetch scap scripts -> run sync-common to fetch MediaWiki -> rebuild l10n cache. Doing this much manual work per app server isn't viable. Giuseppe and I fixed some of this but there's more work to be done.
• Mark submitted a series of patches (principally <https://gerrit.wikimedia.org/r/#/c/152903/>) to create a service IP and Varnish backend for an HHVM app server pool, with Giuseppe and Brandon providing feedback and amending the change. Brandon thinks it looks good and he may be able to deploy it some time next week.
• The patch routes requests that are tagged with a specific cookie to the HHVM backends. Initially, we'll ask you (Wikimedia engineers and technically savvy / adventurous editors) to opt-in to help with testing by setting the cookie explicitly. The next step after that will be to divert a fraction of general site traffic to those backends. When exactly that happens will depend on how many bugs the next round of testing will uncover.
• We'll be adding servers to HHVM pool as we reimage them.
• Tim is looking at modifying the profiling feature of LuaSandbox to work with HHVM. We currently disable it, due to <https://bugzilla.wikimedia.org/show_bug.cgi?id=68413>. (Feedback from users re: how important is this feature to you would be useful).
• Giuseppe and Filippo noticed a memory leak on the HHVM job runner (mw1053). Aaron is trying to isolate it. Tracked in bug <https://bugzilla.wikimedia.org/show_bug.cgi?id=69428>.
• Giuseppe is on vacation for the week of 8/18 - 8/22; Filippo is the point-person for HHVM in TechOps.

2014-08-monthly

We migrated test.wikipedia.org to HHVM in early August and saw very few issues. Giuseppe shared some promising benchmarks. Re-imaging an app server was surprisingly painful, in that Giuseppe and Ori had to perform a number of manual actions to get the server up-and-running, and this sequence of steps was poorly automated. Doing this much manual work per app server isn't viable.

Mark submitted a series of patches to create a service IP and Varnish back-end for an HHVM app server pool, with Giuseppe and Brandon providing feedback and support. The patch routes requests tagged with a specific cookie to the HHVM back-ends. Tech-savvy editors were invited to opt-in to help with testing by setting the cookie explicitly. The next step after that will be to divert a fraction of general site traffic to those back-ends. The exact date will depend on how many bugs the next round of testing uncovers.

Tim is looking at modifying the profiling feature of LuaSandbox to work with HHVM; it is currently disabled.

2014-11-monthly

Rollout is ongoing. The plan is to deploy HHVM on 100% of app servers by Christmas, barring unforeseen issues. Tim Starling and Giuseppe Lavagetto isolated and resolved an issue with OAuth: Authorization header was not available to HHVM via apache_request_headers() due to a mod_proxy_fcgi bug. It was resolved by reinjecting the header via a SetEnvIf directive in the Apache config.

Tim Starling isolated and fixed an API throughput issue (phab:T758). Shelling out to tidy didn’t perform well on HHVM. It was solved by making the tidy extension for PHP compatible with HHVM (at least for essential functionality) and using that instead. This needs to be tested and fully deployed.