Wikimedia Apps/Team/RESTBase services for apps

Jump to: navigation, search

The Reading engineering team is developing a Node.js mobile content service backed by RESTBase to provide content in a form tailored to the needs of the mobile platforms.

General ideas & goals[edit source]

The idea for the services mentioned here is to provide a layer of abstraction on top of various MediaWiki action API and existing RESTBase requests, custom-made for consumption by apps. In other words, they provide a Façade which makes it easy for apps to consume content from Wikipedia. The initial main goal is to improve page load performance.

We want to achieve that through the following approaches:

  • Standard endpoint structure instead of dealing with many query parameters that can be arranged differently; less cache fragmentation.
  • Reduce amount of payload by removing unneeded content.
  • Reduce the need for separate requests by aggregating information from multiple request into fewer requests.
  • Flatten and trim JSON structures. (Again, remove unused data.)
  • Take advantage of Parsoid annotations to improve the quality of the transformations done.
  • Move DOM transformations of page content (currently done client-side) to the server.

Service usage[edit source]

The service endpoints are used by the Android app. Android app users get to use them by default except for usages of zhwiki or when Show images is disabled in the app settings. In those two cases it falls back to using regular api.php endpoints, and some newer features which are only implemented for RESTBase users are automatically disabled. In the app developer settings you can check if RESTBase is enabled and change that if necessary.

The Wikipedia Android app uses this service to get an article's opening section, table of contents, description, lead image URL, and other article information in a single request, followed by another request for the remaining sections. Other endpoints are used to fetch content for link previews and term definitions. The Android app also uses the smart random and the aggregated feed endpoint to retrieve the data needed for the cards of the Explore feed which are not user specific.

The iOS app team is working on using the aggregated feed endpoint for the iOS app. The web team is considering using RB/MCS in the future.

Routes[edit source]

Production routes start with {protocol}://{domain}/api/rest_v1/.

Local dev box routes start with http://localhost:6927/{domain}/v1/ (for MCS directly) or http://localhost:7231/{domain}/v1/ (for RB).

.../page/mobile-sections/{title}[edit source]

.../page/mobile-sections-lead/{title}[edit source]

.../page/mobile-sections-remaining/{title}[edit source]

These three routes are used by the beta Android app when the useRestbase developer option is enabled (which is now enabled by default).

The output has a similar JSON structure to the PHP action=mobileview module, except:

  • mobile-sections:
    • has a top-level object with two properties: lead and remaining. This is an endpoint which gets the contents of the next two endpoints in one single request, which is useful for refreshing saved pages.
    • Examples: Prod | Beta cluster | Labs | Local RB | Local MCS
  • mobile-sections-lead:
    • Is used for the initial page load.
      • Instead of the thumb object to get the URL for the lead image it has a urls property under image. This object contains a hashtable of common lead image widths (640, 800, 1024) pointing to respective URL of the lead image in the size in pixel.
      • If the article has a pronunciation the the pronunciation object has a url string with the fully qualified URL to the pronunciation file.
      • If the article uses one of the Spoken Wikipedia templates the the spoken object has a urls array with the fully qualified URLs to the parts of a recorded audio version of this article.
      • If there are Geo coordinates associated with the article then the geo object will have the latitude and longitude of the place.
      • The sections array includes the information needed to display the lead section and also to build the table of contents. Therefore, it has the section text of the lead section only and the rest of the sections don't include it.
    • Examples: Prod | Beta cluster | Labs | Local RB | Local MCS
  • mobile-sections-remaining:
    • Note that this route's sections array does not include the lead section text since this was already retrieved as part of the lead response.
    • Examples: Prod | Beta cluster | Labs | Local RB | Local MCS
  • For debugging: The HTML content comes from Parsoid. To make it more convenient to debug transformations at a high level here are the respective examples from Parsoid:
  • For comparison, here are the equivalent action=mobileview requests the Android app uses:

Preview: .../page/formatted/{title}[edit source]

Note: new, experimental endpoint. Do not use unless you are one of its developers.

This endpoint is meant to replace the mobile-sections* endpoints. The initial aim is to make this suitable for the web then adopt by the apps.

This would rearrange some things in a backwards-breaking fashion if we did it in mobile-sections, hence the new endpoint. It moves some items from the HTML into JSON. There would be no lead or remaining but it would omit the reference sections, which would be accessible on demand by the next endpoint. More info is coming in phab:T146944.

Preview: .../page/references/{title}/{revId}[edit source]

Note: new, experimental endpoint. Do not use unless you are one of its developers.

Companion endpoint to the formatted endpoint, providing the reference sections.

Examples: Prod | Beta cluster | Labs | Local RB | Local MCS

.../page/summary/{title}[edit source]

This route provides a summary text snippet (generated by the TextExtracts extension), thumbnail info (via PageImages), if a thumbnail is available, and the article language and directionality (RTL or LTR). Used by the apps to generate link previews.

Note: This route is provided directly by RESTBase and is not available in the mobileapps service development branch (i.e., when running the service locally).

Examples: Prod | Beta cluster | Labs | Local RB | Local MCS

For comparison, here are the equivalent action=query requests this endpoint uses: Prod.

.../page/definition/{title}[edit source]

This route provides a set of definitions pulled from the Wiktionary page from the term. (It does not provide the Wiktionary content in full.)

Currently used in the Wikipedia Beta Android app, where users can view a popup with definitions by highlighting a word in the app and choosing the "define" option from the context menu.

Available for English Wiktionary only; rollout to other languages pending based on user engagement.

Note: this endpoint is not for Wikipedia sites, only for Wiktionary.

Examples: Prod | Beta cluster (MCS and Parsoid not setup for some reason) | Labs | Local RB | Local MCS (Wiktionary entry of bar)

.../page/random/{format}[edit source]

MCS provides the title format. All other formats (summary and mobile-sections-lead) are provided by RESTBase. See T132597 (Agree on feed endpoints).

This endpoint tries to provide more interesting pages in its result than a straight random MW API query. It prefers pages with a lead image, WD description, and longer text extract.


.../feed/announcements[edit source]

This endpoint is meant to provide information about surveys and fundraising announcements for the iOS and Android apps only. It is experimental to the extent that it might significantly change or even go away in the future, more likely than other experimental endpoints. Client should code very defensively about the structure and the existence of this endpoint. If a client gets 404s, an exponential backoff strategy may be advisable.

Examples: Prod | Beta cluster | Labs | Local RB | Local MCS

.../feed/featured/{yyyy}/{mm}/{dd}[edit source]

This endpoint provides an aggregation of feed related microservices for one specific day. Note that year has to be exactly four digits, and month and day have to be two digits. Pad with 0 if needed. Earliest year supported is 2016. Example: 2016/07/01.

The response contains the following properties:

  • tfa: featured article (WP languages supported: bg, cs, de, el, en, fa, fr, he, hu, ja, la, no, ur, vi)
  • image: featured image of the day (from Wikimedia Commons)
  • mostread: a list of the previous day's top read articles
  • news: current news, irrespective of day requested. This item is only available for a few wikis right now: da, de, el, en, es, fi, fr, he, ko, no, pl, pt, ru, sv, vi. Latest list and implementation if you want to help us expand it to more languages.

While the other feed microservices are implemented in MCS they are not exposed via RESTBase at this time. Some example URIs to just invoke the microservices locally is in the of the source repo.

Examples: Prod | Beta cluster | Labs | Local RB | Local MCS (Aggregated feed for Febuary 6th, 2017)

For debugging: Local MCS routes of microservices: tfa | image | mostread | news (news in MCS directly is always current, not easy to get to historic content, recent versions of the aggregated RESTBase endpoint try to preserve historical news as much as possible)

.../feed/onthisday/{type}/{mm}/{dd}[edit source]

This endpoint provides information about what event which happened on a specific day and month of the year. Note that month and day have to be two digits. Pad with 0 if needed. Example: selected/07/01. Supported types of events and some examples:

Route usage[edit source]

We have a RESTBase dashboard in Grafana which shows request rates for all individual endpoints. You can choose all the endpoints related to mobile on that graph to get the metrics of how many client requests actually hit RESTBase. The requests are split to several categories:

  • internal - the request came from the WMF cluster or Labs
  • internal_update - it’s an update request from Change-Propagation
  • external - the request came from an external user.

However, for external requests this represent only the cache misses while the vast majority of the requests is served by Varnish.

There's also a Grafana dashboard specifically for the mobile-sections requests.

Development & deployment[edit source]

Source[edit source]

The services are in the following Gerrit repos:

  1. mediawiki/services/mobileapps (diffusion) (GitHub mirror)
  2. mediawiki/services/mobileapps/deploy (diffusion) (GitHub mirror)

The second repo is for deployment purposes. The first repo contains the implementation of the service routes. Both repos are based on the templates provided by our services team.

The Swagger spec can be found in the source repo in the file spec.yaml. This spec must be updated when the output structure is changed since there are automated tests which verify that the output adheres to the spec.

Some settings for IntelliJ are documented for your convenience. You can also use other text editors or IDEs of course. Still, the discussion about run/debug configurations might be applicable for other IDEs, too.

Testing with the Android app[edit source]

The Android app has a Developer settings screen which lets you change the backends used for both MW API calls and RESTBase/MCS calls. If you have a developer flavor of the Android apk then the Developer settings are already enabled. For other flavors you need to enable the Developer settings first. To do so go to Settings, About the Wikipedia app, then tap seven times on the logo. Then you should get a Snackbar saying "You are now a developer!". Once enabled you can tap on the new icon in the top right of the toolbar of the Settings screen. For MCS development you'll want to change the RESTBaseUriFormat to utilize on the the following options described in the following sections. The dialog has example URIs in the hints which can be copy&pasted to the text input form field.

  • Prod: use the production RESTBase servers (the default)
  • Beta cluster settings: (There's no specific entry here but you can get it to work using the following settings:
    • RESTBaseUriFormat (default): %1$s://%2$s/api/rest_v1/
    • mediaWikiBaseUriSupportsLangCode (default): enabled
    • mediaWikiBaseUri (nondefault):
  • Labs: use the labs service
  • Dev: use a local developer machine which runs MCS and possibly RESTBase, too. Note that where it says host you want to replace that with the actual hostname or IP address of the host that's running your MCS services. Some of the features (link previews, aka. summary endpoint, and the aggregated feed) that are implemented directly in RESTBase don't work unless you are pointing to a local RESTBase installation. To use a local RESTBase installation change the port from the MCS port 6927 to the RB port 7231.

Setting up a local RESTBase instance[edit source]

Since there is some interactions between RESTBase and MCS which make it desirable to also run RB locally sometimes (featured feed; hydration of summary data), here are some hints on how to configure RB to work with MCS.

  1. Clone RB
  2. npm install
  3. cp config.example.wikimedia.yaml config.yaml
  4. In your new config.yaml, change all occurrences of cassandra to sqlite
  5. Under default_project, change the host value for mobileapps from to http://localhost:6927
  6. Start MCS (if it isn't already running)
  7. Start RESTBase with node server or npm start in the restbase root directory
  8. RESTBase listens on port 7231 by default

Development on local machine[edit source]

The file in the repo has some great pointers on how to set up and use the service on a dev machine.

MW Vagrant[edit source]

Enable the mobilecontentservice role in MW Vagrant.

vagrant enable-role mobilecontentservice
vagrant provision

The code is located under srv/mobileapps. To restart just the service without having to restart the whole Vagrant instance you can run:

vagrant ssh
sudo service mobileapps restart

Since the Vagrant instance is self-contained you cannot access other servers. If you have a page called Foo in your Vagrant instance you can access it via the following command after sshing into the box:

curl -v localhost:8888/

The log file is logs/mobileapps.log.

Deployment on labs machine[edit source]

phab:T91794 Deploy experimental version of mobile apps content service

The service on is updated and restarted automatically a few minutes after code gets merged.

Troubleshooting on labs machine:

ssh appservice.eqiad.wmflabs
sudo -s
sudo -u nobody /bin/bash
cd /srv/deployment/mobileapps
  • Restart the service:
sudo systemctl restart mobileapps
  • view logs:
tail -f /var/log/syslog | grep -i mobileapps

Beta cluster[edit source]

Similarly to the beta instance on deployment-restbase0[12].deployment-prep.eqiad.wmflabs, there is now also a MCS instance deployment-mcs01.deployment-prep.eqiad.wmflabs.

You can see examples for the various endpoints running in the Beta Cluster listed with each endpoint above. Here's just one example:

Deployment on Production cluster[edit source]

Setup notes[edit source]

The service is deployed on several machines in Service Cluster B.

Deployment process[edit source]

A description of the deployment process we follow.

Deployment schedule[edit source]

Deployment calendar

Deployment logs[edit source]

Every once in a while someone would like to know if patch XYZ has been deployed yet. Lately we note the deployment tag in the Phab task. In addition to that here are a couple of other options to find indications of that.

  • Look for mobileapps in the Server Admin Log (or directly on #wikimedia-operations) then look up the commit message of the mentioned SHA1 in the deploy repo. This option is great for better real-time notification that a new version of MCS got deployed to production.
  • Check out the tags in the source repo: git log --decorate . This happens usually a bit later.