User:N3X15/rewrites/Manual:Mathoid

Group:	Wikimedia Engineering
Team members:	Wikimedia Services
Lead:	no lead

Mathoid is an application which takes various forms of math input and converts it to MathML + SVG or PNG output.

Getting started

Mathoid is a web-service implemented using node.js, often referred to simply as node.

Mathoid setup

Mediawiki-vagrant

Install MediaWiki vagrant and run

vagrant enable role mathoid && sudo vagrant provision

NPM package (Debian Jessie)

sudo apt-get install nodejs nodejs-legacy nodejs-dev npm librsvg2-dev pkg-config git g++
git clone https://github.com/wikimedia/mathoid
cd mathoid
npm install
screen
nohup node server.js

If you want to run npm test you'll need to install the mocha package.

NPM package (Debian Stretch)

sudo apt-get install nodejs nodejs-legacy nodejs-dev npm librsvg2-dev pkg-config git g++
mkdir -p ~/git
cd ~/git
git clone https://github.com/wikimedia/mathoid
cd mathoid
cp config.prod.yaml config.yaml
# edit the config.yaml file to your convenience.
vim config.yaml

If you want to run npm test you'll need to install the mocha package.

To setup mathoid to run as a standalone daemon and start automatically, move the folder to a system folder, and create a "mathoid" user to run the daemon as:

sudo useradd --home=/var/lib/mathoid -M --user-group --system --shell=/usr/sbin/nologin -c "Mathoid for MediaWiki" mathoid
sudo mkdir -p /var/lib/mathoid
sudo chown mathoid:mathoid /var/lib/mathoid

sudo mv ~/git/mathoid /usr/local/lib/mathoid
sudo chown -R root:root /usr/local/lib/mathoid

Create a /etc/systemd/system/mathoid.service file with the following content:

[Unit]
Description=Mediawiki Mathoid Service
Documentation=https://www.mediawiki.org/wiki/Mathoid
Wants=local-fs.target network.target
After=local-fs.target network.target

[Install]
WantedBy=multi-user.target

[Service]
Type=simple
User=mathoid
Group=mathoid
WorkingDirectory=/usr/local/lib/mathoid
ExecStart=/usr/bin/node /usr/local/lib/mathoid/server.js
KillMode=process
Restart=on-success
Nice=10
OOMScoreAdjust=900
PrivateTmp=true
StandardOutput=syslog

And then enable and start the service with the following commands:

sudo systemctl daemon-reload
sudo systemctl enable mathoid
sudo systemctl start mathoid
sudo systemctl status mathoid

NPM package (CentOS 7)

# install dependencies
sudo yum install -y epel-release
sudo yum install -y nodejs npm mocha librsvg2-devel pkgconfig git
sudo yum groupinstall -y 'Development Tools'

# put Mathoid in /etc or wherever else
cd /etc
sudo git clone https://github.com/wikimedia/mathoid
cd mathoid

# npm install and test
sudo npm install 
npm test

# Run server. Or use screen and nohup to run in background 
node server.js

Trouble shooting with CentOS 6.

If you have C++11 related error, install upgraded gcc.

# install upgraded gcc
sudo yum -y install centos-release-scl
sudo yum -y install devtoolset-3-toolchain

use new gcc. (you can back to old gcc with exit.)

# new shell with devtoolset-3
scl enable devtoolset-3 bash 
# you can verify gcc version with "gcc --version"

If you got error about CAIRO_FORMAT_INVALID, edit cairo.h in /usr/include/cairo/cairo.h like below.

/* /usr/include/cairo/cairo.h */  
 typedef enum _cairo_format {
     CAIRO_FORMAT_INVALID   = -1,
     CAIRO_FORMAT_ARGB32    = 0,
     CAIRO_FORMAT_RGB24     = 1,
     CAIRO_FORMAT_A8        = 2,
     CAIRO_FORMAT_A1        = 3,
     //CAIRO_FORMAT_RGB16_565 = 4,
 } cairo_format_t;

/* comment out below */
// typedef enum _cairo_format {
//    CAIRO_FORMAT_ARGB32,
//    CAIRO_FORMAT_RGB24,
//    CAIRO_FORMAT_A8,
//    CAIRO_FORMAT_A1
    /* The value of 4 is reserved by a deprecated enum value.
     * The next format added must have an explicit value of 5.
//    CAIRO_FORMAT_RGB16_565 = 4,
    */
//} cairo_format_t;

Ubuntu package (unmaintained)

sudo add-apt-repository ppa:physikerwelt/mathoid
sudo apt-get update
sudo apt-get install mathoid

Scientific Linux 7.2

Instructions based on SL 7.2 minimal install. Should be the same for Red Hat Enterprise Linux (RHEL), CentOS and clones. Important install latest Mathoid version, at time of writing 0.6.3 and corresponding node.js version, at time of writing 4.x LTS branch. Especially VM/Server needs direct internet connection during installation. Internet access via proxy did not work. Using node.js from repository did not work for Debian 8.5, Ubuntu 16.4, Ubuntu 14.4, Fedora 24 alpha and SL7 + EPEL.

First install prerequisites

Run the following as root. Disable SL7 fastbug repo (optional).

yum -y update
yum -y install gcc-c++ make epel-release librsvg2-devel git

Install Node.js and Mathoid

Run the following as root. Get node.js repo from nodesource. Note one may want to first save and inspect the script before executing it as root.

curl -sL https://rpm.nodesource.com/setup_4.x | bash -
yum -y install nodejs

Switch to user account

git config --get remote.origin.url
npm install mathoid@0.6.3
cd node_modules/mathoid
cp config.dev.yaml config.yaml
vi config.yaml # configure as needed

Test

To test run Mathoid as user with

./server.js
CTRL + C # to end

Update firewall, setup security (e.g. SELINUX), setup to run as service, proxy server to access internet, etc... One should be able to install mocha through npm, if one wants.

LocalSettings.php

Mediawiki requires changes to LocalSettings.php to work with Mathoid, and also needs the Math extension installed.

wfLoadExtension( 'Math' );
$wgMathValidModes[] = 'mathml';
$wgDefaultUserOptions['math'] = 'mathml';
// IP and port of Mathoid server
$wgMathMathMLUrl = 'http://127.0.0.1:10042/';

Configuring

Configuration for Mathoid is done within YAML files that are loaded by the cli.js script, or by the service daemon (server.js). Examples of how WMF configures Mathoid for development (config.dev.yaml) and deployment (config.prod.yaml) are included within the repository for reference.

To configure Mathoid with the most basic settings, suitable for a small wiki or wiki farm:

Remove the config.yaml symlink, if it exists (rm config.yaml)
Copy config.dev.yaml to config.yaml.(cp config.dev.yaml config.yaml)
Open config.yaml with your favorite text editor and change any of the default settings, as desired.

The default configuration in config.dev.yaml assumes that each wiki that accesses Mathoid uses /w as its $wgScriptPath

Settings

Root
Setting Key	Default (dev)	Default (prod)	Notes
num_workers	0	ncpu	Defines how many workers to spawn. ncpu will spawn one worker per CPU core. 0 will ensure only one worker spawns and that it stays within the main thread.
worker_heap_limit_mb	250	500	Maximum allowed size of the v8 heap in MB before the worker is gracefully restarted.
logging.level	trace	warn	Logging level used. trace spams debug messages and up, info (?) logs informational messages and up, warn logs warnings and up, and error (?) logs errors only.
logging.streams	(Empty list)	WMF Logging server	A list of logging endpoints, using gelf. See Logging Configuration table for options. Leave list empty to disable logging to external sources.
metrics	(Commented out)	WMF Statsd server	A statsd endpoint. Optional, comment out to disable.
services	mathoid service	mathoid service	A list of available services. Only one (Mathoid) currently defined. See #Service.

Logging Configuration
Setting Key	Default (prod)	Notes
type	gelf	Type of logging endpoint. Can be any of: gelf stdout stderr debug syslog
host	logstash1003.eqiad.wmnet	Hostname or IP of the logging server, if using gelf
port	12201	Port of the logging server, if using gelf.
uris	(Not used)	Comma-delimited list of possible server hostnames and ports. One will be randomly chosen for each worker. Example: logstash1001.eqiad.wmnet:12201,logstash1003.eqiad.wmnet:12201

Metrics Configuration
Setting Key	Default (prod)	Notes
type	statsd	Type of metrics endpoint. Can be any of: log statsd
host	statsd.eqiad.wmnet	Hostname or IP of the statsd or logging server.
port	8125	Port of the statsd or logging server.
name	(Not Used)	Referenced in the log for reference purposes


Setting Key	Default (dev)	Default (prod)	Notes
name	methoid	mathoid	DO NOT CHANGE. Used in settings-runner.
module	./app.js	./app.js	DO NOT CHANGE. Used in settings-runner.
conf.interface	(Unspecified)	(Unspecified)	Define interface to listen on. Use localhost to only listen on localhost.
conf.port	10044	10042	Port that settings-runner listens on.
conf.spec	(Unspecified)	(Unspecified)	YAML spec to use. spec.yaml by default.
conf.cors	*	*	Allows cross-domain access to the API, * by default. To disable, use false. Can take a hostname to restrict cross-domain requests to that domain.
conf.csp	(Unspecified)	(Unspecified)	Defines content of CSP headers. Uncomment to disable CSP.
conf.proxy	(Unspecified)	(Unspecified)	URI (including protocol) of outgoing proxy.
conf.no_proxy_list	(Unspecified)	(Unspecified)	List of domains which shouldn't be proxied.
conf.log_header_whitelist	(Unspecified)	(Unspecified)	List of incoming HTTP Request headers to log. Defaults to: cache-control content-length content-type if-match user-agent x-request-id
conf.svg	true	true	Render to SVG
conf.img	true	true	Render to JPG
conf.png	true	true	Render to PNG
conf.speech	true	false	Render speech data
conf.texvcinfo	true	true	Render with texvc? (Unclear)
conf.speech_on	true	false	Enable speech
conf.speech_config.semantics	true	true
conf.speech_config.domain	mathspeak	mathspeak
conf.speech_config.style	default	default
conf.speech_config.semantic	true	true
conf.speech_config.minSTree	false	false
conf.speech_config.speakText	true	true
conf.speech_config.enrich	true	true
conf.mj_config.extensions	TeX/color,TeX/mediawiki-texvc,TeX/mhchem	(Same)	Extensions of MathJax to enable
conf.mj_config.MathJax.menuSettings.semantics	true	true
conf.mj_config.MathJax.SVG.font	TeX	TeX	Font to use in SVGs
conf.mj_config.MathJax.TeX.noErrors.disabled	true	true	Suppress TeX errors
conf.no_check	false	true
conf.dpi	180	180	DPI to use when rendering
conf.svgo	false	false	Use svgo to minify SVGs
conf.mwapi_req.method	post	post	Method to use when sending requests to MW API. Can be post or get.
conf.mwapi_req.uri \|https://{{domain}}/w/api.php	http://api.svc.eqiad.wmnet/w/api.php	MW API endpoint URI to use. Can use replacement macros.
conf.mwapi_req.headers.user-agent	{{user-agent}}	{{user-agent}}	User-Agent to send to MW API. Can use replacement macros.
conf.mwapi_req.headers.host	N/A	{{request.params.host}}	Host header to send to MW API. Can use replacement macros.
conf.mwapi_req.body	{{ default(request.query, {}) }}	{{ default(request.query, {}) }}	Body of the POST. Can use replacement macros.

{{macro}} variables and functions
Variable	Meaning
domain	Host header sent to the server
request.query	GET query parameters (like ?a=b)
request.headers	Object containing HTTP headers sent to the server.
request.method	Request HTTP method (GET or POST)
request.body	Body of request
path	Path referenced in the request.
default(value, default)	If value is null or undefined, returns default

Make sure both mwapi_req and restbase_req are pointing to the correct URIs.

Performance

With speech support, performance may be terrible, if there are many complex equations. you can turn off speech support.

speech: false

Testing

After installation you can test your installation via

 curl -d 'q=E=mc^2' localhost:10042

the expected output is:

{"mml":"<math xmlns=\"http://www.w3.org/1998/Math/MathML\" display=\"block\" alttext=\"upper E equals m c squared\">\n  <mi>E</mi>\n  <mo>=</mo>\n  <mi>m</mi>\n  <msup>\n    <mi>c</mi>\n    <mrow class=\"MJX-TeXAtom-ORD\">\n      <mn>2</mn>\n    </mrow>\n  </msup>\n</math>","speakText":"upper E equals m c squared","svg":"<?xml version=\"1.0\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" style=\"vertical-align: -0.167ex; \" width=\"9ex\" height=\"2.167ex\" viewBox=\"0 -904.8 3885.6 940.8\" xmlns=\"http://www.w3.org/2000/svg\" role=\"math\" aria-labelledby=\"MathJax-SVG-1-Title MathJax-SVG-1-Desc\">\n<title id=\"MathJax-SVG-1-Title\">Equation</title>\n<desc id=\"MathJax-SVG-1-Desc\">upper E equals m c squared</desc>\n<defs aria-hidden=\"true\">\n<path stroke-width=\"10\" id=\"E1-MJMATHI-45\" d=\"M492 213Q472 213 472 226Q472 230 477 250T482 285Q482 316 461 323T364 330H312Q311 328 277 192T243 52Q243 48 254 48T334 46Q428 46 458 48T518 61Q567 77 599 117T670 248Q680 270 683 272Q690 274 698 274Q718 274 718 261Q613 7 608 2Q605 0 322 0H133Q31 0 31 11Q31 13 34 25Q38 41 42 43T65 46Q92 46 125 49Q139 52 144 61Q146 66 215 342T285 622Q285 629 281 629Q273 632 228 634H197Q191 640 191 642T193 659Q197 676 203 680H757Q764 676 764 669Q764 664 751 557T737 447Q735 440 717 440H705Q698 445 698 453L701 476Q704 500 704 528Q704 558 697 578T678 609T643 625T596 632T532 634H485Q397 633 392 631Q388 629 386 622Q385 619 355 499T324 377Q347 376 372 376H398Q464 376 489 391T534 472Q538 488 540 490T557 493Q562 493 565 493T570 492T572 491T574 487T577 483L544 351Q511 218 508 216Q505 213 492 213Z\"></path>\n<path stroke-width=\"10\" id=\"E1-MJMAIN-3D\" d=\"M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z\"></path>\n<path stroke-width=\"10\" id=\"E1-MJMATHI-6D\" d=\"M21 287Q22 293 24 303T36 341T56 388T88 425T132 442T175 435T205 417T221 395T229 376L231 369Q231 367 232 367L243 378Q303 442 384 442Q401 442 415 440T441 433T460 423T475 411T485 398T493 385T497 373T500 364T502 357L510 367Q573 442 659 442Q713 442 746 415T780 336Q780 285 742 178T704 50Q705 36 709 31T724 26Q752 26 776 56T815 138Q818 149 821 151T837 153Q857 153 857 145Q857 144 853 130Q845 101 831 73T785 17T716 -10Q669 -10 648 17T627 73Q627 92 663 193T700 345Q700 404 656 404H651Q565 404 506 303L499 291L466 157Q433 26 428 16Q415 -11 385 -11Q372 -11 364 -4T353 8T350 18Q350 29 384 161L420 307Q423 322 423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 181Q151 335 151 342Q154 357 154 369Q154 405 129 405Q107 405 92 377T69 316T57 280Q55 278 41 278H27Q21 284 21 287Z\"></path>\n<path stroke-width=\"10\" id=\"E1-MJMATHI-63\" d=\"M34 159Q34 268 120 355T306 442Q362 442 394 418T427 355Q427 326 408 306T360 285Q341 285 330 295T319 325T330 359T352 380T366 386H367Q367 388 361 392T340 400T306 404Q276 404 249 390Q228 381 206 359Q162 315 142 235T121 119Q121 73 147 50Q169 26 205 26H209Q321 26 394 111Q403 121 406 121Q410 121 419 112T429 98T420 83T391 55T346 25T282 0T202 -11Q127 -11 81 37T34 159Z\"></path>\n<path stroke-width=\"10\" id=\"E1-MJMAIN-32\" d=\"M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z\"></path>\n</defs>\n<g stroke=\"currentColor\" fill=\"currentColor\" stroke-width=\"0\" transform=\"matrix(1 0 0 -1 0 0)\" aria-hidden=\"true\">\n <use xlink:href=\"#E1-MJMATHI-45\" x=\"0\" y=\"0\"></use>\n <use xlink:href=\"#E1-MJMAIN-3D\" x=\"1046\" y=\"0\"></use>\n <use xlink:href=\"#E1-MJMATHI-6D\" x=\"2107\" y=\"0\"></use>\n<g transform=\"translate(2990,0)\">\n <use xlink:href=\"#E1-MJMATHI-63\" x=\"0\" y=\"0\"></use>\n <use transform=\"scale(0.707)\" xlink:href=\"#E1-MJMAIN-32\" x=\"619\" y=\"583\"></use>\n</g>\n</g>\n</svg>","img":"<img src=\"file.svg\" style=\"vertical-align: -0.167ex; margin-left: 0ex; margin-right: 0ex; margin-bottom: 1px; margin-top: 1px; width:9ex; height:2.167ex;\" alt=\"upper E equals m c squared\" />","success":true,"log":"success","sanetex":"E=mc^{2}"}

you can use the option type=mml for mathml input.

Credits for

Gabriel Wicke
Frédéric Wang
Anton Grbin
Davide P. Cervone
Marko Obrovac

Troubleshooting

app.js is not found

in the config file change services->mathoid->module from app.js to mathoid/app.js

Development

Code review happens in Gerrit. See Gerrit/Getting started and ping the parsoid team in #mediawiki-parsoid.

The most recent development version is available from

https://github.com/wikimedia/mathoid

MathOnWikipedia meeting

Improving the rendering of chemical formulae (slide deck)