MediaWiki

User:DKinzler (WMF)/RESTbase development setup

< User:DKinzler (WMF)

Daniel’s field nodes on running RESTbase for development. YMMV.

Installation:

edit

See https://www.mediawiki.org/wiki/RESTBase/Installation

Daniel’s config: https://phabricator.wikimedia.org/P49564

In config.yaml, adjust the domain part of the path specs to match any domain (around line 45 and 67): 

    paths:
        /{domain}/{api:v1}

Under x-sub-request-filters and for apiUriTemplate as well as parsoidHost change the domain to be the address of mediawiki as seen from inside the container. That’s probably the host’s IP. Try host.docker.internal: 

    /action:
         	x-modules:
           	- path: sys/action.js
             		options:
               			# XXX Check API URL!
               			apiUriTemplate: "http://host.docker.internal:8888/w/api.php"
               			# XXX Check the base RESTBase URI
               			baseUriTemplate:"<nowiki>{{'http://{domain}:7231/{domain}/v1'}}</nowiki>"
     /page_save:
         	x-modules:
           	- path: sys/page_save.js
     /parsoid:
         	x-modules:
           	- path: sys/parsoid.js
                     	options:
                       	parsoidHost: <nowiki>http://host.docker.internal:8888/w/rest.php</nowiki>
                       	response_cache_control: 's-maxage=0, max-age=86400'

To only have one worker to deal with (useful for attaching a debugger): 

# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
num_workers: 0

# Number of milliseconds to wait for a heartbeat from worker before killing
# and restarting a worker. 'false' means disabling the heartbeat feature.
worker_heartbeat_timeout: false

I also changed this in this logging section at the bottom, for sanity: 

logging:
    name: restbase
    level: warn

Run in plain node container

edit

docker run --rm -v `pwd`:/usr/src/app -w /usr/src/app -p 7231:7231 -p 9229:9229 -p 9230:9230  --name restbase  node:16 node --inspect=0.0.0.0:9229 server.js

NOTE: we tell the node debugger to listen on port 9229, but we also  publish port 9230. That’s because restbase uses ServiceRunner, which (depending on setup) starts worker processes for restbase, which will then use the next ports, starting at 9230.

Setting num_workers: 0 will make everything run in a single process, so RESTbase will use debug port 9229.

The output should look something like this:

~/code/restbase> docker run --rm -v `pwd`:/usr/src/app -w /usr/src/app -p 7231:7231 -p 9229:9229 -p 9230:9230  --name restbase  node:16 node --inspect=0.0.0.0:9229 server.js

Should show something like: 

Debugger listening on ws://0.0.0.0:9229/748a4d7e-24de-49c6-b8c6-d30a0dd8b4eb

For help, see: https://nodejs.org/en/docs/inspector

{"name":"restbase","hostname":"208c55b610fe","pid":1,"level":50,"levelPath":"error/metrics","msg":"No such metrics client: 'undefined'","time":"2023-07-17T09:59:12.281Z","v":0}

{"name":"restbase","hostname":"208c55b610fe","pid":1,"level":40,"levelPath":"warn/startup","msg":"listening on *:7231","time":"2023-07-17T09:59:12.780Z","v":0}

Verify that RESTbase is up and running:

edit

check that the port is exposed: docker container ls. Shows 0.0.0.0:7231->7231/tcp

curl http://localhost:7231/localhost/v1/ returns {"items":["page","transform"]}. Yay!

NOTE: The path must contain the hostname used in $wgServer! In the past, this has to also include any port number present in $wgServer, with the “:” encoded: e.g. http://localhost:7231/localhost%3A8888/v1/

Call Parsoid endpoint

edit

curl http://localhost:7231/localhost/v1/page/html/Main_Page

^-- this should do something in restbase, even if it fails to connect to mediawiki to actually get the html.

NOTE: use --head to only see the response header

NOTE: until recently, the domain in the path had to include the port as well, as in /localhost%3A8888/.

Run Mocha Tests

edit

If a RESTbase container is already running

> docker exec -it restbase /bin/bash

To start a container for running the tests:

docker run -it --rm -v `pwd`:/usr/src/app -w /usr/src/app -p 9229:9229 --name restbase-mocha node:16 bash

Then, inside the container, run:

./node_modules/.bin/mocha test

To ensure you are running on a clean slate (with SQLite), use export RB_SQLITE_FILE=`mktemp -t sqlite.XXXXXXXXXX`.

If you want to debug the tests, add --inspect-brk=0.0.0.0:9229.

NOTE: Mocha tests use their own config, config.yaml is ignored! Mocha tests will hit the beta cluster!

If Mocha tests for parsoid transformations are failing because they require HTTPS (T350353), you have to adjust the parsoid host URL. In config.fullstack.test.yaml, change the parsoid host URL so it uses HTTPS:

host: https://parsoid-external-ci-access.beta.wmflabs.org/w/rest.php.

If you are getting TLS errors, you may have to first run export NODE_TLS_REJECT_UNAUTHORIZED=0.

Not that RESTbase will connect to this host but it will send Host: en.wikipedia.beta.wmflabs.org in the headers. To re-create this on the command line, use curl -k -H "Host: en.wikipedia.beta.wmflabs.org" https://parsoid-external-ci-access.beta.wmflabs.org/w/rest.php/.

WebStorm Setup

edit

In WebStorm, create a new run configuration of the type “Attach to Node.js/Chrome”.

Set the port to the appropriate number (9229 or 9230, depending on setup).

Set up path mappings to match the volume mount point in the docker container, e.g. map the project directory to /usr/src/app.

Using Blubber

edit

In .pipeline/blubber.yaml:

add this for the development and test variants: 

  runs:
      insecurely: true

Reason: sqlite needs to be able to write to ./db.sqlite3

To enable debugging, modify the entrypoint of the development and test variants:

entrypoint: [node, "--inspect=0.0.0.0:9229", server.js]

Generate docker file, using blubber:

edit

blubber .pipeline/blubber.yaml development | docker build --tag parsoid-dev --file - .

Run RESTbase from the image blubber defined:

edit

docker run --rm --expose 7231 --expose 9229 -p 7231:7231 -p 9229:9229 parsoid-dev

NOTE: looks like the dockerfile generated by blubber doesn’t have an EXPOSE directive. Why?...

HOW can I expose ports in blubber?

DOH: wasted an hour because I put the parameters after the image name, instead of before…

WebStorm setup

edit

In WebStorm, create a new run configuration of the type “Attach to Node.js/Chrome” [TBD]

Retrieved from "https://www.mediawiki.org/w/index.php?title=User:DKinzler_(WMF)/RESTbase_development_setup&oldid=6572061"