Talk:RESTBase/Archive 2

We welcome your comments and technical feedback about the new RESTBase API. For bugs and issues, the recommended way to report issues is https://phabricator.wikimedia.org/

Previous discussion is at Talk:RESTBase/Archive.

RESTBase acts as a data store, and storage is either sqlite or cassandra.

What I can't find anywhere in the documentation is the retention policy of the stored data.

After some months or years of usage, the sqlite file is in the order of Gigabytes. Quite unmanageable. I simply stopped the service, deleted the sqlite file and started it again, and everything seems to work. However, is there any purge or maintenance process of the data that I should run periodically to clean up unneeded data? I mean, stopping the service, manually removing the file and starting again is not particularly clean (although doable from a shell script run from cron). However, is it safe? As I said, apparently everything works, but I'd like to be sure it's safe to do, or if there are more clean ways to remove old data.

I want to migrate the store backend to cassandra, and I fear I'll have the same problems, data storage getting bigger without limit, and the only way to clean it would be to delete the storage entirely and let it recreate itself.

WMF should have bigger storage needs for all its wikis. How is WMF doing this? Do they really have terabytes of data in their cassandra cluster? Is all the data really needed?

Looks like the intention is to not clean old data, and if possible, store the HTML of every possible revision.

See: https://phabricator.wikimedia.org/T97692#1531999

In May, 2022, WMF is using 43 TB of storage for their cassandra instances according to grafana.

After revisiting this, and finding the sqlite db file being 7.5GB of size, I decided to rename the db file to _old and restart the service.

I tried to reload a page that contained math formulas and surprisingly the formulas stoped working. Opening the image URL in the browser resulted in a json error telling it wasn't able to find a file. Quite shocking. The error disappeared once I edited the page.

Apparently, math images aren't regenerated on the fly, which means the database should never be purged, otherwise all math formulas will start failing until the page is edited+previewed, or force-purged.

I didn't expect Restbase database being used to store PNG images. That looks pretty inefficient.

Does it come along with MediaWiki 1.35 or should it be installed separately? If it doesn't come along with MediaWiki 1.35 is it mandatory for VisualEditor to work properly or we can skip its installation?

I had to enable VisualEditor to make it work on 1.35.0

I was able to get

rest.php/v1/page/Main_page/bare and rest.php/v1/page/Main_page to work

But rest.php/v1/page/Main_page/html fails.

It gives the following error

{

"messageTranslations":

{"en":"Unable to fetch Parsoid HTML"},

"httpCode":500,

"httpReason":"Internal Server Error"

}

What should I do to make html endpoint to work?

+1 to the question "What should I do to make html endpoint to work?"

I've tried to install apache Cassandra on latest Debian (bullseye), to use as a storage backend of RESTBase instead of sqlite, to see if it improves storage usage. It has been an impossible task and very time consuming. So please, don't try yourself unless you're adventurous and have a lot of time to spend on this!

I started by installing the latest Cassandra (version 4). Cassandra works. After configuring RESTBase to use Cassandra as the backend, RESTBase doesn't start. Or rather, it starts, but it never reaches the "startup finished" state... It connects to Cassandra, initializes everything, and then it seems to get stuck somewhere to the point that it never listens to the configured port.

Then I decided to install version 2.2.x of Cassandra, to see if it was a compatibility problem. WMF seems to be using 2.2.x (OMG very old version!). But neither it or 3.11.x works on latest Debian. Cassandra won't start. It fails with errors that require editing several config files, removing configurations not supported on recent Java versions... and after making it run, looks like some tools like nodetool will refuse to work, with cryptic errors that apparently are caused by not using Java 8.

Yes, Cassandra 2.x and 3.x only supports Java 8, which is not included by default in Debian. Even cassandra 4.x says support for Java 11 is experimental.

Is it worth the hassle? No

</rant>

I understand installing a RESTBase server will help with dirty differences. I keep getting these 'return' symbols and errors in conversion when switching from wikieditor to visualeditor and from what I read I'd need RESTBase server to help with that.

Is it possible to set up as a Lambda service in Amazon AWS instead of having a fully devoted server? i.e. does anyone have any pointers as to whether that should be possible (I assume HTTP API Gateway + Lambda function + Amazon Keyspaces for Apache Cassandra)?

Any pointer would be really helpful.

Thank you!

First, I had the same issue as the user below, "No storage group configured for localhost". I overcame that, to be able to start the server, but it doesn't work.

Attempting to start VisualEditor fails with "Error loading data from server: HTTP 200. Would you like to retry?" And I get the following error in RESTBase (i have edited out my domain):

{"name":"restbase","hostname":"bones","pid":51160,"level":40,"err":{"message":"404: not_found#route","name":"restbase","stack":"HTTPError: 404: not_found#route

   at HyperSwitch._request (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:268:19)

   at _createFilteredHandler (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:170:26)

   at tryCatcher (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/util.js:16:23)

   at /usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/method.js:15:34

   at handlerWrapper (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:422:37)

   at next (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:408:42)

   at Object.module.exports [as filter] (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/filters/http.js:24:16)

   at handlerWrapper (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:420:27)

   at /usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:426:28

   at HyperSwitch._filteredRequest (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:171:19)

   at HyperSwitch.(anonymous function) [as post] (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:460:21)

   at mwUtil.getSiteInfo.then (/usr/home/dave/restbase-1.0.0/sys/parsoid.js:88:22)

   at tryCatcher (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/util.js:16:23)

   at Promise._settlePromiseFromHandler (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:517:31)

   at Promise._settlePromise (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:574:18)

   at Promise._settlePromise0 (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:619:10)","status":404,"body":{"type":"not_found#route","title":"Not found.","internalURI":"/fake.example.net/sys/events/","method":"post","depth":2},"root_req":{"method":"get","uri":"/fake.example.net/v1/page/html/MediaWiki_Server/24?redirect=false","headers":{"user-agent":"VisualEditor-MediaWiki/1.32.0","x-client-ip":"::1","x-forwarded-for":"::1","x-request-id":"8b8d1910-a45b-11e9-a949-65e3af41582b","x-request-class":"internal"}},"request_id":"8b8d1910-a45b-11e9-a949-65e3af41582b","api_path":"/fake.example.net/v1/page/html/{title}/{revision}","levelPath":"warn/bg-updates"},"msg":"404: not_found#route","time":"2019-07-12T04:14:32.084Z","v":0}

Can anyone help me out? I'm really stumped here and I just want my VisualEditor to work right.

I attempted to fix the error by editing projects/example.yaml with the "events" config from the other examples. Now I'm getting this:

{"name":"restbase","hostname":"bones","pid":51532,"level":50,"message":"500: internal_error","res":{"name":"HTTPError","message":"500: internal_error","status":500,"body":{"type":"internal_error","description":"TypeError: Cannot set property 'etag' of undefined"},"innerError":{"requestName":"get_from_backend"},"headers":{"access-control-allow-origin":"*","access-control-allow-methods":"GET,HEAD","access-control-allow-headers":"accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding","access-control-expose-headers":"etag","x-content-type-options":"nosniff","x-frame-options":"SAMEORIGIN","referrer-policy":"origin-when-cross-origin","x-xss-protection":"1; mode=block","content-security-policy":"default-src 'none'; frame-ancestors 'none'","x-content-security-policy":"default-src 'none'; frame-ancestors 'none'","x-webkit-csp":"default-src 'none'; frame-ancestors 'none'","cache-control":"private, max-age=0, s-maxage=0, must-revalidate","x-request-id":"f6006bc0-a45c-11e9-899d-cd6b4b45393b","server":"bones"}},"stack":"TypeError: Cannot set property 'etag' of undefined

   at contentReq.then.then (/usr/home/dave/restbase-1.0.0/sys/parsoid.js:511:30)

   at tryCatcher (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/util.js:16:23)

   at Promise._settlePromiseFromHandler (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:517:31)

   at Promise._settlePromise (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:574:18)

   at Promise._settlePromise0 (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:619:10)

   at Promise._settlePromises (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:699:18)

   at Promise._fulfill (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:643:18)

   at Promise._resolveCallback (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:437:57)

   at Promise._settlePromiseFromHandler (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:529:17)

   at Promise._settlePromise (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:574:18)

   at Promise._settlePromise0 (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:619:10)

   at Promise._settlePromises (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:699:18)

   at Promise._fulfill (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:643:18)

   at Promise._settlePromise (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:587:21)

   at Promise._settlePromise0 (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:619:10)

   at Promise._settlePromises (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:699:18)","latency":288,"root_req":{"method":"get","uri":"/fake.example.net/v1/page/html/MediaWiki_Server/24?redirect=false","headers":{"user-agent":"VisualEditor-MediaWiki/1.32.0","x-client-ip":"::1","x-forwarded-for":"::1","x-request-id":"f6006bc0-a45c-11e9-899d-cd6b4b45393b","x-request-class":"internal"}},"request_id":"f6006bc0-a45c-11e9-899d-cd6b4b45393b","levelPath":"error/request","msg":"500: internal_error","time":"2019-07-12T04:24:38.556Z","v":0}

I also get that error when trying to use curl as suggested by the install docs. First the curl works, and then subsequent attempts return that error.

Again, any help is truly appreciated.

I am also having the Fresh install in another RHEL7 server, And it fails with following ERROR.

{"name":"restbase","hostname":"linapp3050","pid":24490,"level":50,"message":"worker died, restarting","worker_pid":24598,"exit_code":1,"levelPath":"error/service-runner/master","msg":"worker died, restarting","time":"2019-07-15T13:02:21.353Z","v":0}

@Anu8791You need to look at the lines before to know what caused the worker died.

@2601:19A:8380:7EF7:99D5:B7D2:FF2:5F71 I've got the same error after setting up a fresh install (without WMF-like URLs now). By logging at trace level, I can see restbase succeeds in talking with parsoid as far as it gets the html content of the page but then I've got this 500 error with the same message (and at line 511 in sys/parsoid.js too).

Hey there,

at the moment, I'm trying to install a new RESTBase service for testing purposes. I'm using openSUSE Leap 15.0 for my environment.

After downloading the RESTBase components over git and installing all necessary packages trough npm install (with the same steps as they're documented at the RESTBase article), my system only throws the following error messages over and over again until I kill the RESTBase process:

{"name":"restbase","hostname":"linux-p2rf","pid":134906,"level":60,"err":{"message":"No storage group configured for localhost","name":"Error","stack":"Error: No storage group configured for localhost\n    at DB._resolveStorageGroup (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:88:19)\n    at DB._tableName (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:96:27)\n    at DB.createTable (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:155:43)\n    at RBSQLite.createTable (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/index.js:37:22)\n    at tryCatcher (/var/lib/restbase/restbase/node_modules/bluebird/js/release/util.js:16:23)\n    at /var/lib/restbase/restbase/node_modules/bluebird/js/release/method.js:15:34\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:422:37)\n    at next (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:408:42)\n    at Object.module.exports [as filter] (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/filters/validator.js:272:12)\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:420:27)\n    at next (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:408:42)\n    at Object.module.exports [as filter] (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/filters/metrics.js:16:12)\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:420:27)\n    at /var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:426:28\n    at P.resolve.then (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:229:24)\n    at tryCatcher (/var/lib/restbase/restbase/node_modules/bluebird/js/release/util.js:16:23)"},"stack":"Error: No storage group configured for localhost\n    at DB._resolveStorageGroup (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:88:19)\n    at DB._tableName (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:96:27)\n    at DB.createTable (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:155:43)\n    at RBSQLite.createTable (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/index.js:37:22)\n    at tryCatcher (/var/lib/restbase/restbase/node_modules/bluebird/js/release/util.js:16:23)\n    at /var/lib/restbase/restbase/node_modules/bluebird/js/release/method.js:15:34\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:422:37)\n    at next (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:408:42)\n    at Object.module.exports [as filter] (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/filters/validator.js:272:12)\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:420:27)\n    at next (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:408:42)\n    at Object.module.exports [as filter] (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/filters/metrics.js:16:12)\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:420:27)\n    at /var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:426:28\n    at P.resolve.then (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:229:24)\n    at tryCatcher (/var/lib/restbase/restbase/node_modules/bluebird/js/release/util.js:16:23)","levelPath":"fatal/startup","msg":"Message not supplied","time":"2019-06-11T20:44:22.888Z","v":0}

At this point, I've killed the RESTBase process. The error message above loops over and over again.

{"name":"restbase","hostname":"linux-p2rf","pid":134865,"level":30,"levelPath":"info/service-runner/master","msg":"master shutting down, killing workers","time":"2019-06-11T20:44:23.853Z","v":0}

{"name":"restbase","hostname":"linux-p2rf","pid":134865,"level":30,"levelPath":"info/service-runner/master","msg":"Exiting master","time":"2019-06-11T20:44:23.872Z","v":0}

Any ideas?

Looks like a breaking change was made in the most recent version of the sqlite module. Take the "storage_groups" bit of the sample config at github.com/wikimedia/restbase-mod-table-sqlite/blob/master/test/test_client.conf.yaml and add it into the "table" bit of your restbase config. Seemed to work for me.

The above suggestion worked for me.

I just added the following to "the table bit" of my config:

storage_groups:
    - name: default.group
      domains: /./

PS: there's an issue for it here: https://phabricator.wikimedia.org/T227486

I've already added it there : RESTBase/Installation#Configuration ;)

I'm learning the RESTBase API, and noticed some idiosyncracies about using today's date. Many calls fail if I try to pull today's data, until the update job runs, and some calls like edits/aggregate only have data for the previous month. What's the recommended way to handle this? Should I hardcode matching behavior into my API calls, in other words only pull aggregate edit counts for the previous calendar month, given some margin of safety like 1 day in order to guarantee that the data will be populated? Or should I just retry time intervals, shifting them backwards until the call succeeds? Is the update job schedule documented somewhere canonical?

Is there a page like API:Client code for RESTBase? I've been working on a client library and would love to look at other implementations for inspiration. Also, I'd like my library to be discoverable when the time comes.

Duplicate of Topic:Uo3kkmmop1jj9vhw. Follow-up there.

Using mediawiki 1.31 and Math extension REL_31

Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":)

If i change it to allow code 0:

 diff --git a/MathRestbaseInterface.php b/MathRestbaseInterface.php
 index 2dc5d45..5355177 100644
 --- a/MathRestbaseInterface.php
 +++ b/MathRestbaseInterface.php
 @@ -364,7 +364,7 @@ class MathRestbaseInterface {
          */
         public function evaluateRestbaseCheckResponse( $response ) {
                 $json = json_decode( $response['body'] );
 -               if ( $response['code'] === 200 ) {
 +               if ( $response['code'] === 200 || $response['code'] === 0) {
                         $headers = $response['headers'];
                         $this->hash = $headers['x-resource-location'];
                         $this->success = $json->success;
 

I get

Failed to parse (Conversion error. Server ("https://wikimedia.org/api/rest_") reported: "Cannot get mml. Server problem.")

Is the RESTBase server down?