Wikimedia Apps/Team/RESTBase services for apps

The Product Infrastructure team maintains a set of Node.js services intended to provide wiki content in a form tailored to the needs of the Wikimedia mobile platforms, and in particular the Wikimedia Apps.

These services include:

Mobile Content Service (MCS): A service built to replace the apps' usage of the MediaWiki action API mobileview module and to incorporate page content transforms that were previously being performed in JavaScript on the client post-page load. MCS is slated for service decommissioning in July 2023.

Page Content Service (PCS): The successor to the Mobile Content Service, intended to make a fuller set of page content available to a variety of clients through the Wikimedia REST API.
Wikifeeds: A service that provides featured wiki content in JSON format via the Wikimedia REST API.

In production, each of these services is consumed by RESTBase, which is in turn exposed to the public via the REST API.

OpenAPI/Swagger documentation: REST API (prod), MCS/PCS (labs), Wikifeeds (labs)

General ideas & goals

The idea for the services mentioned here is to provide a layer of abstraction on top of various MediaWiki action API and existing RESTBase requests, custom-made for consumption by apps. In other words, they provide a Façade which makes it easy for apps to consume content from Wikipedia. The initial main goal is to improve page load performance.

We want to achieve that through the following approaches:

Standard endpoint structure instead of dealing with many query parameters that can be arranged differently; less cache fragmentation.
Reduce amount of payload by removing unneeded content.
Reduce the need for separate requests by aggregating information from multiple request into fewer requests.
Flatten and trim JSON structures. (Again, remove unused data.)

Take advantage of Parsoid annotations to improve the quality of the transformations done.
Move DOM transformations of page content (currently done client-side) to the server.

Service usage

The service endpoints are used by the Android app. Android app users get to use them by default except for usages of zhwiki or when Show images is disabled in the app settings. In those two cases it falls back to using regular api.php endpoints, and some newer features which are only implemented for RESTBase users are automatically disabled. In the app developer settings you can check if RESTBase is enabled and change that if necessary.

The Wikipedia Android app uses this service to get an article's opening section, table of contents, description, lead image URL, and other article information in a single request, followed by another request for the remaining sections. Other endpoints are used to fetch content for link previews and term definitions. The Android app also uses the smart random and the aggregated feed endpoint to retrieve the data needed for the cards of the Explore feed which are not user specific.

The iOS app team is working on using the aggregated feed endpoint for the iOS app. The web team is considering using RB/MCS in the future.

Monitoring

We have a RESTBase dashboard in Grafana which shows request rates for all individual endpoints. You can choose all the endpoints related to mobile on that graph to get the metrics of how many client requests actually hit RESTBase. The requests are split to several categories:

internal - the request came from the WMF cluster or Labs
internal_update - it’s an update request from Change-Propagation
external - the request came from an external user.

However, for external requests this represents only the cache misses while the vast majority of the requests are served by Varnish.

There's also a Grafana dashboard specifically for the mobile-sections requests.

Development & deployment

Source

The services are in the following Gerrit repos:

source: (Gerrit) (diffusion) (GitHub mirror)
deploy: (Gerrit) (diffusion) (GitHub mirror) - nowadays unused and archived, see phab:T247012

The second repo is for deployment purposes. The first repo contains the implementation of the service routes. The source repo is based on the ServiceTemplateNode project provided by the Services team.

The Swagger spec can be found in the source repo in the file spec.yaml. This spec must be updated when the output structure is changed since there are automated tests which verify that the output adheres to the spec.

Some settings for IntelliJ are documented for your convenience. You can also use other text editors or IDEs of course. Still, the discussion about run/debug configurations might be applicable for other IDEs, too.

Testing with the Android app

The Android app has a Developer settings screen which lets you change the backends used for both MW API calls and RESTBase/MCS calls. If you have a developer flavor of the Android apk then the Developer settings are already enabled. For other flavors you need to enable the Developer settings first. To do so go to Settings, About the Wikipedia app, then tap seven times on the logo. Then you should get a Snackbar saying "You are now a developer!". Once enabled you can tap on the new icon in the top right of the toolbar of the Settings screen. For MCS development you'll want to change the RESTBaseUriFormat to utilize on the following options described in the following sections. The dialog has example URIs in the hints which can be copy&pasted to the text input form field.

Prod: use the production RESTBase servers (the default)
Beta cluster settings: (There's no specific entry here but you can get it to work using the following settings:
- RESTBaseUriFormat (default): %1$s://%2$s/api/rest_v1/
- mediaWikiBaseUriSupportsLangCode (default): enabled
- mediaWikiBaseUri (nondefault): https://wikipedia.beta.wmflabs.org
Labs: use the labs service
Dev: use a local developer machine which runs MCS and possibly RESTBase, too. Note that where it says host you want to replace that with the actual hostname or IP address of the host that's running your MCS services. Some of the features (link previews, aka summary endpoint, and the aggregated feed) that are implemented directly in RESTBase don't work unless you are pointing to a local RESTBase installation. To use a local RESTBase installation change the port from the MCS port 6927 to the RB port 7231.

Setting up a local RESTBase instance

Since there is some interactions between RESTBase and MCS which make it desirable to also run RB locally sometimes (featured feed; hydration of summary data), here are some hints on how to configure RB to work with MCS.

Clone RB
npm install
cp config.example.wikimedia.yaml config.yaml
In your new config.yaml, change all occurrences of cassandra to sqlite
Under default_project, change the host value for mobileapps from http://mobileapps.wmflabs.org to http://localhost:6927
You may also want to consider adding the following at the top of the file to run RB in a debugger:

num_workers: 0

Start MCS (if it isn't already running)
Start RESTBase with node server or npm start in the restbase root directory
RESTBase listens on port 7231 by default.
Test URI: http://localhost:7231/en.wikipedia.org/v1/page/mobile-sections/Cassini%E2%80%93Huygens

Setting up a local Parsoid instance

To test new Parsoid patches:

Clone Parsoid repo
npm install
cp config.example.yaml config.yaml

In your new config.yaml, you may want to edit the mwApis: section. For example to hook it up with a few production Wikipedias:

        mwApis:
        - # English WP
          uri: 'https://en.wikipedia.org/w/api.php'
          domain: 'en.wikipedia.org'  # optional
        - # French WP
          uri: 'https://fr.wikipedia.org/w/api.php'
          domain: 'fr.wikipedia.org'
        - # Test WP
          uri: 'https://test.wikipedia.org/w/api.php'
          domain: 'test.wikipedia.org'

You may also want to consider adding the following at the top of the file to run Parsoid in a debugger:

num_workers: 0

Start Parsoid with npm start in the Parsoid root directory.
Test URI example (note the v3): http://localhost:8000/en.wikipedia.org/v3/page/html/Foobar/798652007

Now it's time to update the MCS config file.

If you want MCS to talk directly talk to Parsoid instead through RESTBase then use these settings: Change the uri in restbase_req of MCS' config.dev.yaml
```
      restbase_req:
        method: '{{request.method}}'
        uri: http://localhost:8000/{{domain}}/v3/{+path}
```

If you want to go through a local RB instance then use instead:

      restbase_req:
        method: '{{request.method}}'
        uri: http://localhost:7231/{{domain}}/v1/{+path}

Start MCS with npm start in the MCS folder
Test URI: http://localhost:6927/en.wikipedia.org/v1/page/mobile-sections/Foobar/798652007

Development on local machine

The README.md file in the repo has some great pointers on how to set up and use the service on a dev machine.

MW Vagrant

Enable the mobilecontentservice role in MW Vagrant.

vagrant enable-role mobilecontentservice
vagrant provision

The code is located under srv/mobileapps. To restart just the service without having to restart the whole Vagrant instance you can run:

vagrant ssh
sudo service mobileapps restart

Since the Vagrant instance is self-contained you cannot access other servers. If you have a page called Foo in your Vagrant instance you can access it via the following command after sshing into the box:

curl -v localhost:8888/dev.wiki.local.wmftest.net/v1/page/mobile-sections-lead/Foo

The log file is logs/mobileapps.log.

Standalone Cloud VPS testing instances

The mobileapps and wikifeeds services both have standalone, automatically updated testing instances available in Cloud VPS.

The instances are mobileapps.eqiad.wmflabs and wikifeeds.eqiad.wmflabs in the Cloud VPS Mobile project, exposed publicly at https://mobileapps.wmflabs.org and https://wikifeeds.wmflabs.org respectively.

Each is updated and restarted automatically a few minutes after code gets merged.

Troubleshooting on Cloud VPS instance:

ssh mobileapps.eqiad.wmflabs
sudo -s
sudo -u nobody /bin/bash
cd /srv/mobileapps

Restart the service:

sudo systemctl restart mobileapps

view logs:

tail -f /var/log/syslog | grep -i mobileapps

See setup instructions for a testing instance like these.

Beta cluster

Similarly to the beta instance on deployment-restbase0[12].deployment-prep.eqiad.wmflabs, there is now also a MCS instance deployment-mcs01.deployment-prep.eqiad.wmflabs.

You can see examples for the various endpoints running in the Beta Cluster listed with each endpoint above. Here's just one example: https://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/mobile-sections/Dog.

Deployment on Production cluster

The mobileapps service (supporting MCS and PCS) is deployed on Service Cluster B, but will soon be deployed on Kubernetes by way of the deployment pipeline. The wikifeeds service is already deployed to k8s on the pipeline.

Deployment process

A description of the legacy deployment process we follow for mobileapps.

Deployment schedule

Deployment calendar

Deployment logs

Every once in a while someone would like to know if patch XYZ has been deployed yet. Lately we note the deployment tag in the Phab task. In addition to that here are a couple of other options to find indications of that.

Look for mobileapps in the Server Admin Log (or directly on #wikimedia-operations ^connect) then look up the commit message of the mentioned SHA1 in the deploy repo. This option is great for better real-time notification that a new version of MCS got deployed to production.
Check out the tags in the source repo: git log --decorate . This happens usually a bit later.