Configure Restbase to work with the Math extension

To install mathoid and restbase to work with the math extension the following steps are required:

Mathoid

npm i mathoid
cd node_modules/mathoid

edit config.yaml as follows

# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
# Use 'ncpu' to run as many workers as there are CPU units
num_workers: ncpu

# Log error messages and gracefully restart a worker if v8 reports that it
# uses more heap (note: not RSS) than this many mb.
worker_heap_limit_mb: 500

# Logger info
logging:
  level: warn

# Statsd metrics reporter
metrics:
  type: log
  #host: localhost
  #port: 8125

services:
  - name: mathoid
    # a relative path or the name of an npm package, if different from name
    module: ./app.js
    # optionally, a version constraint of the npm package
    # version: ^0.4.0
    # per-service config
    conf:
      port: 10042
      # interface: localhost # uncomment to only listen on localhost
      # more per-service config settings
      svg: true
      img: true
      png: true
      texvcinfo: true
      speech: true
      speech_on: false
      chem: true
      no_check: true

run mathoid as a screen session

screen
node server.js -c config.yaml

testing mathoid

Now the following test

curl -d 'q=E=mc^2' localhost:10042/mml

should return

<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
  <semantics>
    <mrow>
      <mi>E</mi>
      <mo>=</mo>
      <mi>m</mi>
      <msup>
        <mi>c</mi>
        <mn>2</mn>
      </msup>
    </mrow>
    <annotation encoding="application/x-tex">E=mc^2</annotation>
  </semantics>
</math>

RESTBase

npm i restbase
cd node_modules/restbase

edit config.yaml as follows

paths:
  /{api:v1}:
    x-modules:
        # swagger options, overriding the shared ones from the merged specs (?)
      - spec:
          info:
            version: 1.0.0-beta
            title: Formulasearchengine REST API
            description: >
                This API aims to provide a preview of new features reagarding
                Math rendering and search, supposed to be integrated to
                the [wikimedia restbase api](https://wikimedia.org/api/rest_v1/?doc).

                ### High-volume access
                  - Don't perform more than 500 requests/s to this API.
                  - Set a unique `User-Agent` header that allows us to contact you
                    quickly. Email addresses or URLs of contact pages work well.

            termsOfService: https://wikimediafoundation.org/wiki/Terms_of_Use
            contact:
              name: the Wikimedia Services team
              url: http://mediawiki.org/wiki/RESTBase
            license:
              name: Apache2
              url: http://www.apache.org/licenses/LICENSE-2.0

          securityDefinitions: &wp/content-security/1.0.0
            mediawiki_auth:
              description: Checks permissions using MW api
              type: apiKey
              in: header
              name: cookie
              x-internal-request-whitelist:
                - /http:\/\/[a-zA-Z0-9\.]+\/w\/api\.php/
          # Override the base path for host-based (proxied) requests. In our case,
          # we proxy https://{domain}/api/rest_v1/ to the API.
          x-host-basePath: /api/rest_v1
          x-route-filters:
            - path: ./lib/normalize_title_filter.js
              options:
                redirect_cache_control: '{{options.purged_cache_control}}'
          paths:
            /media:
              x-modules:
                - path: v1/mathoid.yaml
                  options: '{{options.mathoid}}'
            /testing:
              x-modules:
                - path: ../mathpipe.yaml
                 # options: '{{options.mathoid}}'
        options: '{{options}}'

  /{api:sys}:
    x-modules:
      - spec:
          paths:
            /mathoid:
              x-modules:
                - path: sys/mathoid.js
                  options: '{{options.mathoid}}'
            /table:
              x-modules:
                - path: sys/table.js
                  options:
                    conf: '{{options.table}}'
            /key_value:
              x-modules:
                - path: sys/key_value.js
            /key_rev_value:
              x-modules:
                - path: sys/key_rev_value.js
            /key_rev_latest_value:
              x-modules:
                - path: sys/key_rev_latest_value.js
            /post_data: &sys_post_data
              x-modules:
                - path: sys/post_data.js
            /events:
              x-modules:
                - path: sys/events.js
                  options: '{{options.events}}'
        options: '{{options}}'

run restbase as a screen session

screen
node server.js -c config.yaml

testing restbase

Navigate to http://localhost:8081/wikimedia.org/v1/?doc you should be able to enter a texvc formula to the check command POST endpoing i.e. /media/math/check/{type} (See screenshot 1)[1] In the response headers you should find something like

"x-resource-location": "4c0004393a88f350a93bcef62106d556c7fc827b"

If you copy the value 4c0004393a88f350a93bcef62106d556c7fc827b and paste it to the GET endpoint /media/math/render/{format}/{hash} as value you should be able to see the PNG image if you select that as an output format. (See screenshot 2)[2]

Configuration Example for arbitrary domain in RESTBase setup

If you want to use domain other than wikimedia.org, you can follow below example;

mathoid/config.yaml

# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
# Use 'ncpu' to run as many workers as there are CPU units
num_workers: 0

# Log error messages and gracefully restart a worker if v8 reports that it
# uses more heap (note: not RSS) than this many mb.
worker_heap_limit_mb: 250

# Logger info
logging:
  level: trace
#  streams:
#  # Use gelf-stream -> logstash
#  - type: gelf
#    host: logstash1003.eqiad.wmnet
#    port: 12201

# Statsd metrics reporter
metrics:
  #type: log
  #host: localhost
  #port: 8125

services:
  - name: mathoid
    # a relative path or the name of an npm package, if different from name
    module: ./app.js
    # optionally, a version constraint of the npm package
    # version: ^0.4.0
    # per-service config
    conf:
      port: 10044
      # interface: localhost # uncomment to only listen on localhost
      # more per-service config settings
      # the location of the spec, defaults to spec.yaml if not specified
      # spec: ./spec.template.yaml
      # allow cross-domain requests to the API (default '*')
      #cors: '*'
      # to disable use:
      # cors: false
      # to restrict to a particular domain, use:
      # cors: restricted.domain.org
      # content for the CSP headers
      # csp: false  # uncomment this line to disable sending them
      # URL of the outbound proxy to use (complete with protocol)
      # proxy: http://my.proxy.org:8080
      # the list of domains for which not to use the proxy defined above
      # no_proxy_list:
      #   - domain1.com
      #   - domain2.org
      # the list of incoming request headers that can be logged; if left empty,
      # the following headers are allowed: cache-control, content-length,
      # content-type, if-match, user-agent, x-request-id
      # log_header_whitelist:
      #   - cache-control
      #   - content-length
      #   - content-type
      #   - if-match
      #   - user-agent
      #   - x-request-id
      # list of enabled renders
      svg: true
      img: true
      png: true #new feature
      speech: true #new feature
      texvcinfo: true
      speech_on: true
      no_check: false
      dpi: 180
      svgo: false
      # the user agent to use when issuing requests
      # user_agent: service-template-node
      # the template used for contacting the MW API
      #mwapi_req:
      #  method: post
      #  uri: https://{{domain}}/w/api.php
      #  headers:
      #    user-agent: '{{user-agent}}'
      #  body: '{{ default(request.query, {}) }}'
      # the template used for contacting RESTBase
      #restbase_req:
      #  method: '{{request.method}}'
      #  uri: https://{{domain}}/api/rest_v1/{+path}
      #  query: '{{ default(request.query, {}) }}'
      #  headers: '{{request.headers}}'
      #  body: '{{request.body}}'

restbase/config.yaml

services:
  - name: restbase
    module: hyperswitch
    conf:
      port: 7231
      salt: secret
      default_page_size: 125
      user_agent: RESTBase
      ui_name: RESTBase
      ui_url: https://www.mediawiki.org/wiki/RESTBase
      ui_title: RESTBase docs
      spec:
        x-request-filters:
          - path: lib/security_response_header_filter.js
        x-sub-request-filters:
          - type: default
            name: http
            options:
            allow:
              - pattern: http://localhost/api.php
                forward_headers: true
              - pattern: http://localhost:8000
                forward_headers: true
              - pattern: http://localhost:10044
                forward_headers: true
              - pattern: /^https?:\/\//
        paths:
          /{domain:localhost}: &arbitary_domain
            x-modules:
              - path: projects/MR.yaml
                options:
                  action:
                    # XXX Check API URL!
                    apiUriTemplate: http://localhost/api.php
                    # XXX Check the base RESTBase URI
                    baseUriTemplate: "{{'http://localhost:7231/{domain}/v1'}}"
                  parsoid:
                    # XXX Check Parsoid URL!
                    host: http://localhost:8000
                  mathoid:
                    # XXX Check Mathoid URL!
                    host: http://localhost:10044
                  table:
                    backend: sqlite
                    dbname: db.sqlite3
                    pool_idle_timeout: 20000
                    retry_delay: 250
                    retry_limit: 10
                    show_sql: false
          /{domain:your.some.domain}: *arbitary_domain

# Finally, a standard service-runner config.
info:
  name: restbase

logging:
  name: restbase
  level: info

restbase/project/MR.yaml

# will work with Mathoid on any {domain}
paths:
  /{api:v1}:
    x-modules:
        # swagger options, overriding the shared ones from the merged specs (?)
      - spec:
          info:
            version: 1.0.0
            title: Wikimedia REST API
            description: Welcome to your RESTBase API.
          #securityDefinitions: &wp/content-security/1.0.0
          #  mediawiki_auth:
          #    description: Checks permissions using MW api
          #    type: apiKey
          #    in: header
          #    name: cookie
          #    x-internal-request-whitelist:
          #      - /http:\/\/[a-zA-Z0-9\.]+\/w\/api\.php/
          # Override the base path for host-based (proxied) requests. In our case,
          # we proxy https://{domain}/api/rest_v1/ to the API.
          #x-host-basePath: /api/rest_v1
          x-route-filters:
            - path: ./lib/normalize_title_filter.js
              options:
                redirect_cache_control: '{{options.purged_cache_control}}'
          paths:
            /media:
              x-modules:
                - path: v1/mathoid.yaml
                  options: '{{options.mathoid}}'
            #/testing:
            #  x-modules:
            #    - path: ../mathpipe.yaml
                 # options: '{{options.mathoid}}'
            /page:
              x-modules:
                - path: v1/content.yaml
                  options:
                    purged_cache_control: '{{options.purged_cache_control}}'
                - path: v1/common_schemas.yaml # Doesn't really matter where to mount it.
            /transform:
              x-modules:
                - path: v1/transform.yaml
        options: '{{options}}'

  /{api:sys}:
    x-modules:
      - spec:
          paths:
            /mathoid:
              x-modules:
                - path: sys/mathoid.js
                  options: '{{options.mathoid}}'
            /table:
              x-modules:
                - path: sys/table.js
                  options:
                    conf: '{{options.table}}'
            /key_value:
              x-modules:
                - path: sys/key_value.js
            /key_rev_value:
              x-modules:
                - path: sys/key_rev_value.js
            /key_rev_latest_value:
              x-modules:
                - path: sys/key_rev_latest_value.js
            /post_data: &sys_post_data
              x-modules:
                - path: sys/post_data.js
            /events:
              x-modules:
                - path: sys/events.js
                  options: '{{options.events}}'
            /page_revisions:
              x-modules:
                - path: sys/page_revisions.js
            /action:
              x-modules:
                - path: sys/action.js
                  options: '{{options.action}}'
            /page_save:
              x-modules:
                - path: sys/page_save.js
            /parsoid:
              x-modules:
                - path: sys/parsoid.js
                  options:
                    parsoidHost: '{{options.parsoid.host}}'
                    response_cache_control: '{{options.purged_cache_control}}'
        options: '{{options}}'

restbase/v1/mathoid.yaml

# Mathoid - math formula rendering service :simply replaced /wikimedia.org/sys to /{domain}/sys
tags:
  - name: Math
    description: formula rendering
paths:
  /math/check/{type}:
    post:
      tags: ['Math']
      summary: Check and normalize a TeX formula.
      description: |
        Checks the supplied TeX formula for correctness and returns the
        normalised formula representation as well as information about
        identifiers. Available types are tex and inline-tex. The response
        contains the `x-resource-location` header which can be used to retrieve
        the render of the checked formula in one of the supported rendering
        formats. Just append the value of the header to `/media/math/{format}/`
        and perform a GET request against that URL.

        Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
      produces:
        - application/json
      parameters:
        - name: type
          in: path
          description: The input type of the given formula; can be tex or inline-tex
          type: string
          required: true
          enum:
            - tex
            - inline-tex
            - chem
        - name: q
          in: formData
          description: The formula to check
          type: string
          required: true
      responses:
        '200':
          description: Information about the checked formula
        '400':
          description: Invalid type
          schema:
            $ref: '#/definitions/problem'
        default:
          description: Error
          schema:
            $ref: '#/definitions/problem'
      x-monitor: true
      x-amples:
        - title: Mathoid - check test formula
          request:
            params:
              domain: wikimedia.org
              type: tex
            body:
              q: E=mc^{2}
          response:
            status: 200
            headers:
              content-type: /^application\/json/
              x-resource-location: /.+/
              cache-control: 'no-cache'
            body:
              success: true
              checked: /.+/
      x-request-handler:
        - get_from_sys:
            request:
              method: post
              uri: /{domain}/sys/mathoid/check/{type}
              headers: '{{ request.headers }}'
              body: '{{ request.body }}'

  /math/formula/{hash}:
    get:
      tags: ['Math']
      summary: Get a previously-stored formula
      description: |
        Returns the previously-stored formula via `/media/math/check/{type}` for
        the given hash.

        Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
      produces:
        - application/json
      parameters:
        - name: hash
          in: path
          description: The hash string of the previous POST data
          type: string
          required: true
          minLength: 1
      responses:
        '200':
          description: Information about the checked formula
        '404':
          description: Data for the given hash cannot be found
          schema:
            $ref: '#/definitions/problem'
        default:
          description: Error
          schema:
            $ref: '#/definitions/problem'
      x-monitor: false
      x-request-handler:
        - get_from_sys:
            request:
              method: get
              uri: /{domain}/sys/mathoid/formula/{hash}
              headers: '{{ request.headers }}'

  /math/render/{format}/{hash}:
    get:
      tags: ['Math']
      summary: Get rendered formula in the given format.
      description: |
        Given a request hash, renders a TeX formula into its mathematic
        representation in the given format. When a request is issued to the
        `/media/math/check/{format}` POST endpoint, the response contains the
        `x-resource-location` header denoting the hash ID of the POST data. Once
        obtained, this endpoint has to be used to obtain the actual render.

        Stability: [stable](https://www.mediawiki.org/wiki/API_versioning#Stable).
      produces:
        - image/svg+xml
        - application/mathml+xml
        - image/png
      parameters:
        - name: format
          in: path
          description: The output format; can be svg or mml
          type: string
          required: true
          enum:
            - svg
            - mml
            - png
        - name: hash
          in: path
          description: The hash string of the previous POST data
          type: string
          required: true
          minLength: 1
      responses:
        '200':
          description: The rendered formula
        '404':
          description: Unknown format or hash ID
          schema:
            $ref: '#/definitions/problem'
        default:
          description: Error
          schema:
            $ref: '#/definitions/problem'
      x-monitor: false
      x-setup-handler:
        - init_svg:
            uri: /{domain}/sys/key_value/mathoid.svg
            body:
              keyType: string
              valueType: string
        - init_mml:
            uri: /{domain}/sys/key_value/mathoid.mml
            body:
              keyType: string
              valueType: string
        - init_png:
            uri: /{domain}/sys/key_value/mathoid.png
            body:
              keyType: string
              valueType: blob
      x-request-handler:
        - check_storage:
            request:
              method: get
              uri: /{domain}/sys/key_value/mathoid.{$.request.params.format}/{$.request.params.hash}
              headers:
                cache-control: '{{ cache-control }}'
            catch:
              status: 404
            return_if:
              status: '2xx'
            return:
              status: 200
              headers: "{{ merge({ 'cache-control': options.cache-control }, check_storage.headers) }}"
              body: '{{ check_storage.body }}'
        - postdata:
            request:
              uri: /{domain}/sys/mathoid/formula/{request.params.hash}
        - mathoid:
            request:
              method: post
              uri: /{domain}/sys/mathoid/render/{request.params.format}
              headers:
                content-type: application/json
                x-resource-location: '{{ request.params.hash }}'
              body: '{{postdata.body}}'

Apache proxy config : /etc/httpd/conf.d/restbase.conf

#
# Restbase
#

<VirtualHost *:80>
          ProxyPreserveHost On

          ProxyPass /your.some.domain/ http://localhost:7231/your.some.domain/
          ProxyPassReverse /your.some.domain/ http://localhost:7231/your.some.domain/

          ProxyPass /localhost/ http://localhost:7231/localhost/
          ProxyPassReverse /localhost/ http://localhost:7231/localhost/

          # MediaWiki is located here:
          DocumentRoot /var/www/html
 </VirtualHost>

you can test the APIs at http://your.some.domain/your.some.domain/v1/

setup apache proxy

To load the images from the same location where the main wiki site is hosted a simple apache proxy can be used. Enable apache mod proxy ... and then add a config file along the lines of this file

cat /etc/apache2/sites-available/restbase.conf

<VirtualHost *:80>
    ServerName  restbase.formulasearchengine.com
    ServerAlias api.formulasearchengine.com png.formulasearchengine.com
    ServerAdmin wiki@physikerwelt.de
    ProxyPreserveHost On
    ProxyPass /v1/ http://localhost:8081/wikimedia.org/v1/
    ProxyPass /wikimedia.org/v1/ http://localhost:8081/wikimedia.org/v1/
</VirtualHost>

a2ensite restbase.conf

You can use letsencrypt to also generate a HTTPS version for that site.

Helpful commands to check the state of the database

Clean cache

run cqlsh

use "local_group_default_T_mathoid_svg";
truncate data;
truncate meta;
use "local_group_default_T_mathoid_mml";
truncate data;
truncate meta;
use "local_group_default_T_mathoid_input";
truncate data;
truncate meta;
use "local_group_default_T_mathoid_png";
truncate data;
truncate meta;

Browse the data

Task retrieve the input given a key ... in this example we'll use 0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d as key run cqlsh

use "local_group_test_T_mathoid_input";
SELECT * from data WHERE key = '0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d' and "_domain" = 'wikimedia.org';

 _domain       | key                                      | tid                                  | _del | content-location | content-sha256 | content-type     | headers                             |latestTid | tags | value
 ---------------+------------------------------------------+--------------------------------------+------+------------------+----------------+------------------+-------------------------------------+-----------+------+----------------------------------------------------------------------------------------------------------------------------------------------------------------
 wikimedia.org | 0f7fa7bb7d214c60f8d2f0b8d356c41bfe5c5d1d | e736aba4-9edc-11e5-927a-301f88a60cb9 | null |             null |           null | application/json | {"content-type":"application/json"} |      null | null | {"q":"=nf(x)\\left({n-1 \\choose k-1}F(x)^{k-1}(1-F(x))^{(n-1)-(k-1)}-\\underbrace {{n-1 \\choose n}F(x)^{n}(1-F(x))^{(n-1)-n}} \\right)","type":"inline-tex"}

Extension:Math/RESTBase

Contents