Jump to content


About this board

We welcome your comments and technical feedback about the new RESTBase API. For bugs and issues, the recommended way to report issues is https://phabricator.wikimedia.org/

Previous discussion is at Talk:RESTBase/Archive.

Data store retention policy

Ciencia Al Poder (talkcontribs)

RESTBase acts as a data store, and storage is either sqlite or cassandra.

What I can't find anywhere in the documentation is the retention policy of the stored data.

After some months or years of usage, the sqlite file is in the order of Gigabytes. Quite unmanageable. I simply stopped the service, deleted the sqlite file and started it again, and everything seems to work. However, is there any purge or maintenance process of the data that I should run periodically to clean up unneeded data? I mean, stopping the service, manually removing the file and starting again is not particularly clean (although doable from a shell script run from cron). However, is it safe? As I said, apparently everything works, but I'd like to be sure it's safe to do, or if there are more clean ways to remove old data.

I want to migrate the store backend to cassandra, and I fear I'll have the same problems, data storage getting bigger without limit, and the only way to clean it would be to delete the storage entirely and let it recreate itself.

WMF should have bigger storage needs for all its wikis. How is WMF doing this? Do they really have terabytes of data in their cassandra cluster? Is all the data really needed?

Ciencia Al Poder (talkcontribs)
Ciencia Al Poder (talkcontribs)

After revisiting this, and finding the sqlite db file being 7.5GB of size, I decided to rename the db file to _old and restart the service.

I tried to reload a page that contained math formulas and surprisingly the formulas stoped working. Opening the image URL in the browser resulted in a json error telling it wasn't able to find a file. Quite shocking. The error disappeared once I edited the page.

Apparently, math images aren't regenerated on the fly, which means the database should never be purged, otherwise all math formulas will start failing until the page is edited+previewed, or force-purged.

I didn't expect Restbase database being used to store PNG images. That looks pretty inefficient.

Reply to "Data store retention policy"
Costas Athan (talkcontribs)

Does it come along with MediaWiki 1.35 or should it be installed separately? If it doesn't come along with MediaWiki 1.35 is it mandatory for VisualEditor to work properly or we can skip its installation?

Akhilkodali (talkcontribs)

I had to enable VisualEditor to make it work on 1.35.0

I was able to get

rest.php/v1/page/Main_page/bare and rest.php/v1/page/Main_page to work

But rest.php/v1/page/Main_page/html fails.

It gives the following error



{"en":"Unable to fetch Parsoid HTML"},


"httpReason":"Internal Server Error"


What should I do to make html endpoint to work?

SkywalkerPL (talkcontribs)

+1 to the question "What should I do to make html endpoint to work?"

Reply to "MediaWiki 1.35"
Ciencia Al Poder (talkcontribs)

I've tried to install apache Cassandra on latest Debian (bullseye), to use as a storage backend of RESTBase instead of sqlite, to see if it improves storage usage. It has been an impossible task and very time consuming. So please, don't try yourself unless you're adventurous and have a lot of time to spend on this!

I started by installing the latest Cassandra (version 4). Cassandra works. After configuring RESTBase to use Cassandra as the backend, RESTBase doesn't start. Or rather, it starts, but it never reaches the "startup finished" state... It connects to Cassandra, initializes everything, and then it seems to get stuck somewhere to the point that it never listens to the configured port.

Then I decided to install version 2.2.x of Cassandra, to see if it was a compatibility problem. WMF seems to be using 2.2.x (OMG very old version!). But neither it or 3.11.x works on latest Debian. Cassandra won't start. It fails with errors that require editing several config files, removing configurations not supported on recent Java versions... and after making it run, looks like some tools like nodetool will refuse to work, with cryptic errors that apparently are caused by not using Java 8.

Yes, Cassandra 2.x and 3.x only supports Java 8, which is not included by default in Debian. Even cassandra 4.x says support for Java 11 is experimental.

Is it worth the hassle? No


Reply to "RESTBase + Cassandra"
Krug Walton (talkcontribs)

I understand installing a RESTBase server will help with dirty differences. I keep getting these 'return' symbols and errors in conversion when switching from wikieditor to visualeditor and from what I read I'd need RESTBase server to help with that.

Is it possible to set up as a Lambda service in Amazon AWS instead of having a fully devoted server? i.e. does anyone have any pointers as to whether that should be possible (I assume HTTP API Gateway + Lambda function + Amazon Keyspaces for Apache Cassandra)?

Any pointer would be really helpful.

Thank you!

Reply to "Serverless RESTBase?"

Fresh install of RESTBase doesn't work

2601:19A:8380:7EF7:99D5:B7D2:FF2:5F71 (talkcontribs)

First, I had the same issue as the user below, "No storage group configured for localhost". I overcame that, to be able to start the server, but it doesn't work.

Attempting to start VisualEditor fails with "Error loading data from server: HTTP 200. Would you like to retry?" And I get the following error in RESTBase (i have edited out my domain):

{"name":"restbase","hostname":"bones","pid":51160,"level":40,"err":{"message":"404: not_found#route","name":"restbase","stack":"HTTPError: 404: not_found#route

   at HyperSwitch._request (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:268:19)

   at _createFilteredHandler (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:170:26)

   at tryCatcher (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/util.js:16:23)

   at /usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/method.js:15:34

   at handlerWrapper (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:422:37)

   at next (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:408:42)

   at Object.module.exports [as filter] (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/filters/http.js:24:16)

   at handlerWrapper (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:420:27)

   at /usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:426:28

   at HyperSwitch._filteredRequest (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:171:19)

   at HyperSwitch.(anonymous function) [as post] (/usr/home/dave/restbase-1.0.0/node_modules/hyperswitch/lib/hyperswitch.js:460:21)

   at mwUtil.getSiteInfo.then (/usr/home/dave/restbase-1.0.0/sys/parsoid.js:88:22)

   at tryCatcher (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/util.js:16:23)

   at Promise._settlePromiseFromHandler (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:517:31)

   at Promise._settlePromise (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:574:18)

   at Promise._settlePromise0 (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:619:10)","status":404,"body":{"type":"not_found#route","title":"Not found.","internalURI":"/fake.example.net/sys/events/","method":"post","depth":2},"root_req":{"method":"get","uri":"/fake.example.net/v1/page/html/MediaWiki_Server/24?redirect=false","headers":{"user-agent":"VisualEditor-MediaWiki/1.32.0","x-client-ip":"::1","x-forwarded-for":"::1","x-request-id":"8b8d1910-a45b-11e9-a949-65e3af41582b","x-request-class":"internal"}},"request_id":"8b8d1910-a45b-11e9-a949-65e3af41582b","api_path":"/fake.example.net/v1/page/html/{title}/{revision}","levelPath":"warn/bg-updates"},"msg":"404: not_found#route","time":"2019-07-12T04:14:32.084Z","v":0}

Can anyone help me out? I'm really stumped here and I just want my VisualEditor to work right.

2601:19A:8380:7EF7:99D5:B7D2:FF2:5F71 (talkcontribs)

I attempted to fix the error by editing projects/example.yaml with the "events" config from the other examples. Now I'm getting this:

{"name":"restbase","hostname":"bones","pid":51532,"level":50,"message":"500: internal_error","res":{"name":"HTTPError","message":"500: internal_error","status":500,"body":{"type":"internal_error","description":"TypeError: Cannot set property 'etag' of undefined"},"innerError":{"requestName":"get_from_backend"},"headers":{"access-control-allow-origin":"*","access-control-allow-methods":"GET,HEAD","access-control-allow-headers":"accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding","access-control-expose-headers":"etag","x-content-type-options":"nosniff","x-frame-options":"SAMEORIGIN","referrer-policy":"origin-when-cross-origin","x-xss-protection":"1; mode=block","content-security-policy":"default-src 'none'; frame-ancestors 'none'","x-content-security-policy":"default-src 'none'; frame-ancestors 'none'","x-webkit-csp":"default-src 'none'; frame-ancestors 'none'","cache-control":"private, max-age=0, s-maxage=0, must-revalidate","x-request-id":"f6006bc0-a45c-11e9-899d-cd6b4b45393b","server":"bones"}},"stack":"TypeError: Cannot set property 'etag' of undefined

   at contentReq.then.then (/usr/home/dave/restbase-1.0.0/sys/parsoid.js:511:30)

   at tryCatcher (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/util.js:16:23)

   at Promise._settlePromiseFromHandler (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:517:31)

   at Promise._settlePromise (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:574:18)

   at Promise._settlePromise0 (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:619:10)

   at Promise._settlePromises (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:699:18)

   at Promise._fulfill (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:643:18)

   at Promise._resolveCallback (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:437:57)

   at Promise._settlePromiseFromHandler (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:529:17)

   at Promise._settlePromise (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:574:18)

   at Promise._settlePromise0 (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:619:10)

   at Promise._settlePromises (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:699:18)

   at Promise._fulfill (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:643:18)

   at Promise._settlePromise (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:587:21)

   at Promise._settlePromise0 (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:619:10)

   at Promise._settlePromises (/usr/home/dave/restbase-1.0.0/node_modules/bluebird/js/release/promise.js:699:18)","latency":288,"root_req":{"method":"get","uri":"/fake.example.net/v1/page/html/MediaWiki_Server/24?redirect=false","headers":{"user-agent":"VisualEditor-MediaWiki/1.32.0","x-client-ip":"::1","x-forwarded-for":"::1","x-request-id":"f6006bc0-a45c-11e9-899d-cd6b4b45393b","x-request-class":"internal"}},"request_id":"f6006bc0-a45c-11e9-899d-cd6b4b45393b","levelPath":"error/request","msg":"500: internal_error","time":"2019-07-12T04:24:38.556Z","v":0}

I also get that error when trying to use curl as suggested by the install docs. First the curl works, and then subsequent attempts return that error.

Again, any help is truly appreciated.

Anu8791 (talkcontribs)

I am also having the Fresh install in another RHEL7 server, And it fails with following ERROR.

{"name":"restbase","hostname":"linapp3050","pid":24490,"level":50,"message":"worker died, restarting","worker_pid":24598,"exit_code":1,"levelPath":"error/service-runner/master","msg":"worker died, restarting","time":"2019-07-15T13:02:21.353Z","v":0}

Echodelta21 (talkcontribs)

@Anu8791You need to look at the lines before to know what caused the worker died.

@2601:19A:8380:7EF7:99D5:B7D2:FF2:5F71 I've got the same error after setting up a fresh install (without WMF-like URLs now). By logging at trace level, I can see restbase succeeds in talking with parsoid as far as it gets the html content of the page but then I've got this 500 error with the same message (and at line 511 in sys/parsoid.js too).

Reply to "Fresh install of RESTBase doesn't work"

New RESTBase installation broken - No storage group configured for localhost

Daxfox2172 (talkcontribs)

Hey there,

at the moment, I'm trying to install a new RESTBase service for testing purposes. I'm using openSUSE Leap 15.0 for my environment.

After downloading the RESTBase components over git and installing all necessary packages trough npm install (with the same steps as they're documented at the RESTBase article), my system only throws the following error messages over and over again until I kill the RESTBase process:

{"name":"restbase","hostname":"linux-p2rf","pid":134906,"level":60,"err":{"message":"No storage group configured for localhost","name":"Error","stack":"Error: No storage group configured for localhost\n    at DB._resolveStorageGroup (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:88:19)\n    at DB._tableName (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:96:27)\n    at DB.createTable (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:155:43)\n    at RBSQLite.createTable (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/index.js:37:22)\n    at tryCatcher (/var/lib/restbase/restbase/node_modules/bluebird/js/release/util.js:16:23)\n    at /var/lib/restbase/restbase/node_modules/bluebird/js/release/method.js:15:34\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:422:37)\n    at next (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:408:42)\n    at Object.module.exports [as filter] (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/filters/validator.js:272:12)\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:420:27)\n    at next (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:408:42)\n    at Object.module.exports [as filter] (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/filters/metrics.js:16:12)\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:420:27)\n    at /var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:426:28\n    at P.resolve.then (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:229:24)\n    at tryCatcher (/var/lib/restbase/restbase/node_modules/bluebird/js/release/util.js:16:23)"},"stack":"Error: No storage group configured for localhost\n    at DB._resolveStorageGroup (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:88:19)\n    at DB._tableName (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:96:27)\n    at DB.createTable (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/lib/db.js:155:43)\n    at RBSQLite.createTable (/var/lib/restbase/restbase/node_modules/restbase-mod-table-sqlite/index.js:37:22)\n    at tryCatcher (/var/lib/restbase/restbase/node_modules/bluebird/js/release/util.js:16:23)\n    at /var/lib/restbase/restbase/node_modules/bluebird/js/release/method.js:15:34\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:422:37)\n    at next (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:408:42)\n    at Object.module.exports [as filter] (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/filters/validator.js:272:12)\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:420:27)\n    at next (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:408:42)\n    at Object.module.exports [as filter] (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/filters/metrics.js:16:12)\n    at handlerWrapper (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:420:27)\n    at /var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:426:28\n    at P.resolve.then (/var/lib/restbase/restbase/node_modules/hyperswitch/lib/hyperswitch.js:229:24)\n    at tryCatcher (/var/lib/restbase/restbase/node_modules/bluebird/js/release/util.js:16:23)","levelPath":"fatal/startup","msg":"Message not supplied","time":"2019-06-11T20:44:22.888Z","v":0}

At this point, I've killed the RESTBase process. The error message above loops over and over again.

{"name":"restbase","hostname":"linux-p2rf","pid":134865,"level":30,"levelPath":"info/service-runner/master","msg":"master shutting down, killing workers","time":"2019-06-11T20:44:23.853Z","v":0}

{"name":"restbase","hostname":"linux-p2rf","pid":134865,"level":30,"levelPath":"info/service-runner/master","msg":"Exiting master","time":"2019-06-11T20:44:23.872Z","v":0}

Any ideas? (talkcontribs)

Looks like a breaking change was made in the most recent version of the sqlite module. Take the "storage_groups" bit of the sample config at github.com/wikimedia/restbase-mod-table-sqlite/blob/master/test/test_client.conf.yaml and add it into the "table" bit of your restbase config. Seemed to work for me.

Metalliqaz (talkcontribs)

The above suggestion worked for me.

I just added the following to "the table bit" of my config:

    - name: default.group
      domains: /./

PS: there's an issue for it here: https://phabricator.wikimedia.org/T227486

Echodelta21 (talkcontribs)
Reply to "New RESTBase installation broken - No storage group configured for localhost"

Documentation for update job schedule?

Adamw (talkcontribs)

I'm learning the RESTBase API, and noticed some idiosyncracies about using today's date. Many calls fail if I try to pull today's data, until the update job runs, and some calls like edits/aggregate only have data for the previous month. What's the recommended way to handle this? Should I hardcode matching behavior into my API calls, in other words only pull aggregate edit counts for the previous calendar month, given some margin of safety like 1 day in order to guarantee that the data will be populated? Or should I just retry time intervals, shifting them backwards until the call succeeds? Is the update job schedule documented somewhere canonical?

Reply to "Documentation for update job schedule?"
Adamw (talkcontribs)

Is there a page like API:Client code for RESTBase? I've been working on a client library and would love to look at other implementations for inspiration. Also, I'd like my library to be discoverable when the time comes.

Reply to "Catalog of clients"

Trouble setting up RESTbase in MediaWiki

This post was hidden by Hoefsldyla (history)
Reply to "Trouble setting up RESTbase in MediaWiki"

https://wikimedia.org/api/rest_v1/ returning code 0 and not 200

Summary by Jdforrester (WMF)

Duplicate of Topic:Uo3kkmmop1jj9vhw. Follow-up there.

Nyet (talkcontribs)

Using mediawiki 1.31 and Math extension REL_31

Failed to parse (MathML with SVG or PNG fallback (recommended for modern browsers and accessibility tools): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":)

If i change it to allow code 0:

 diff --git a/MathRestbaseInterface.php b/MathRestbaseInterface.php
 index 2dc5d45..5355177 100644
 --- a/MathRestbaseInterface.php
 +++ b/MathRestbaseInterface.php
 @@ -364,7 +364,7 @@ class MathRestbaseInterface {
         public function evaluateRestbaseCheckResponse( $response ) {
                 $json = json_decode( $response['body'] );
 -               if ( $response['code'] === 200 ) {
 +               if ( $response['code'] === 200 || $response['code'] === 0) {
                         $headers = $response['headers'];
                         $this->hash = $headers['x-resource-location'];
                         $this->success = $json->success;

I get

Failed to parse (Conversion error. Server ("https://wikimedia.org/api/rest_") reported: "Cannot get mml. Server problem.")

Is the RESTBase server down?