Wikimedia Apps/Team/RESTBase services for apps

The apps team has developed new RESTBase services based on node.js and various node modules. The Wikipedia Android app uses one of these to get an article's opening section, TOC, description, lead image URL, and other article information in one request. Other endpoints are used to fetch content for link previews and term definitions. the mobile content service is enabled for almost all Android Beta app users.


 * Sample URL: https://en.wikipedia.org/api/rest_v1/page/mobile-sections-lead/Albert%20Einstein
 * API documentation: Mobile section of rest_v1 docs
 * Phabricator project: #mobile_content_service

General ideas & goals
The idea for the services mentioned here is to provide a layer of abstraction on top of various MediaWiki action API and existing RESTBase requests, custom-made for consumption by apps. In other words, they provide a Façade which makes it easy for apps to consume content from Wikipedia. The initial main goal is to improve page load performance.

We want to achieve that through the following approaches:
 * Reduce amount of payload by removing unneeded content and using Parsoid.
 * Reduce the need for separate requests by aggregating information from multiple request into fewer requests.
 * Flatten and trim JSON structures. (Again, remove unused data.)


 * Take advantage of Parsoid annotations to improve the quality of the transformations done.
 * Move DOM transformations of page content (currently done client-side) to the server.

Service usage
The service endpoints are used by the Android Beta app. All beta app users get to use them by default except for usages of zhwiki or when  is disabled in the app settings. In those two cases it falls back to using regular api.php endpoints, and some newer features which are only implemented for RESTBase users are automatically disabled. In the app developer settings you can check if RESTBase is enabled and change that if necessary.

Routes
All routes start with.

.../mobile-sections-remaining/{title}
These three routes are used by the beta Android app when the Developer option  is enabled.

The output has a similar JSON structure to the PHP  module, except: The Swagger spec can be found in the source repo in the file. This is a good source to see the actual structure of the output. It would also have to be updated when we significantly changed the structure since there are automated tests which verify that the output adheres to the spec.
 * : has a top-level object with two properties:  and  . This is an endpoint which gets the contents of the next two endpoints in one single request, which is useful for refreshing saved pages.
 * : is used for both the link preview and the initial page load.
 * Instead of the  object to get the URL for the lead image it has a   property under  . This object contains a hashtable of common lead image widths (640, 800, 1024) pointing to respective URL of the lead image in the size in pixel.
 * The  array is a JSON representation of the first infobox in the lead section.
 * The  object shows the text extracts we use for the link preview.
 * If the article has a pronunciation the the  object has a   string with the fully qualified URL to the pronunciation file.
 * If the article uses one of the  templates the the   object has a   array with the fully qualified URLs to the parts of a recorded audio version of this article.
 * If there are Geo coordinates associated with the article then the  object will have the   and   of the place.
 * The  array includes the information needed to display the lead section and also to build the table of contents. Therefore, it has the section text of the lead section only and the rest of the sections don't include it.
 * The  section contains a list of images and videos contained on the page and the metadata displayed in the app. Mainly needs license data. (It was originally used in the   response but since the link preview needs this info, too, it was moved to the   response.)
 * : Note that this route's  array does not include the lead section text since this was already retrieved as part of the lead response.

.../mobile-text/{title}
This route is meant for a new generation lite app; initially targeted for low-powered, older Android devices. The idea is, instead of using a WebView, to use native Android UI components to show the page contents.

This route is currently just using action=mobileview but it's foreseeable that it'll use similar backend calls as the previous routes.

More at T90758.

.../summary/{title}
This route provides a summary text snippet (generated by the TextExtracts extension), thumbnail info (via PageImages), if a thumbnail is available, and the article language and directionality (RTL or LTR). Used by the apps to generate link previews.

Note: This route is provided directly by RESTBase and is not available in the mobileapps service development branch (i.e., when running the service locally). For now, the legacy /mobile-summary route remains in the repo but is unused.

.../definition/{term}
This route provides a set of definitions pulled from the Wiktionary page from the term. (It does not provide the Wiktionary content in full.) Definitions are displayed on a popup the user can access by highlighting a word in the app and choosing the "define" option from the context menu.

Currently enabled in the Android Beta app for English only; rollout to other languages pending based on user engagement.

Source
The services are in the following Gerrit repos:
 * 1) mediawiki/services/mobileapps
 * 2) mediawiki/services/mobileapps/deploy

The second repo is for deployment purposes. The first repo contains the implementation of the service routes. Both repos are based on the templates provided by our services team.

Development on local machine
The README.md file in the repo has some great pointers on how to set up and use the service on a dev machine.

Deployment on labs machine
T91794 Deploy experimental version of mobile apps content service

The service on appservice.wmflabs.org is updated and restarted automatically a few minutes after code gets merged. Here is a simple example for some endpoints: Troubleshooting on labs machine:
 * http://appservice.wmflabs.org/en.m.wikipedia.org/v1/page/mobile-sections/Dilbert
 * http://appservice.wmflabs.org/en.m.wikipedia.org/v1/page/mobile-sections-lead/Dilbert
 * http://appservice.wmflabs.org/en.m.wikipedia.org/v1/page/mobile-sections-remaining/Dilbert
 * Restart the service:
 * view logs:

FYI: Beta cluster
This is more an FYI since this is for the RESTBase framework itself. There is a beta instance on deployment-restbase0[12].deployment-prep.eqiad.wmflabs. It would use the labs instance, appservice.wmflabs.org, to complete the mobile route requests.

Deployment on Production cluster
We build the deploy repo in Ubuntu with Docker in order to ensure that the Node modules have the correct binaries. (We've explored the option of using Boot2docker on OS X but ran into issues with the docker script.) Detailed setup notes for the Ubuntu deployment environment can be found here.

The service is deployed on Service Cluster B ( and  ). The deployment procedures are here: Marko's deployment page

Production deployments are tracked with git tags in the main mobileapps repo. The most recent commit included in each deployment is given a tag adhering to the form  (e.g., ).

Other documentation: Docker + Deployment notes from service template

Some example requests:
 * https://en.wikipedia.org/api/rest_v1/page/mobile-sections/Dilbert
 * https://en.wikipedia.org/api/rest_v1/page/mobile-sections-lead/Dilbert
 * https://en.wikipedia.org/api/rest_v1/page/mobile-sections-remaining/Dilbert
 * https://en.wikipedia.org/api/rest_v1/page/summary/electron
 * https://en.wiktionary.org/api/rest_v1/page/definition/lemon

Deployments
Deployment log (deprecated)

Deployment calendar