Wikimedia Apps/Team/RESTBase services for apps

The apps team is in the process of developing and switching to new RESTBase services based on node.js and various node modules.

General ideas & goals
The idea for the services mentioned here is to provide a layer of abstraction on top of various MediaWiki api.php and other RESTBase requests, custom-made for consumption by apps. In other words, they provide a Façade which makes it easy for apps to consume content from Wikipedia. The initial main goal is to improve page load performance.

We want to achieve that through the following approaches: Another emerging/future goal is:
 * Move DOM transformations of page content (currently done client-side) to the server.
 * Reduce amount of payload by removing unneeded content and using Parsoid.
 * Reduce the need for separate requests by aggregating information from multiple request into fewer requests.
 * Flatten and trim JSON structures. (Again, remove unused data.)
 * Take advantage of Parsoid annotations to improve the quality of the transformations done.

Phabricator project
The Phabricator project for this is: mobile_content_service

Routes
All routes start with.

.../page/html/{title}
This route is currently used by the alpha Android app when experimental page load is enabled. The idea is to call webView.loadUrl on the service URL, and let it stream in the content.

When given an existing wikipedia title it responds with HTML page content with two embedded JSON data blocks for metadata:
 * is at the beginning and should contain only what's needed to show things above the fold.
 * is at the end of the payload, mainly for metadata we don't immediately need. Currently it contains some gallery metadata (images and videos)

TODO: Arguably, the toc entry could be moved from the first to the second block. I don't think it makes a big performance impact. So, for now we can keep it there to keep the service simpler. This route executes multiple requests to backend services in parallel via promises:
 * : the main HTML payload
 * This is the page content we get from Parsoid (RESTBase), plus some modifications.
 * : general page metadata -> goes to
 * This data comes from MediaWiki API: This is currently still using action=mobileview. (TODO: Will need to use different backend api requests to be leaner. No need to request page content here, too. We only need the JSON parts.)
 * : other page metadata ->
 * This data comes from MediaWiki API ; with up to two follow-up requests: one for images, another one for videos.

The HTML output also adds links to a CSS and a JavaScript file, which means CSS and JS are coming from the service now instead of being bundled directly with the app. This is easier to get working on the Android app since everything comes from the same source. For subsequent page loads CSS and JS should be cached on the client.

There are currently two alert dialogs triggered by the JS bundle when the DOM is done loading:
 * One generic DOMLoaded event to be sent over the JS bridge to notify the app that the DOM is loaded. This one is probably obsolete for the new model, and we should be able to get rid of it (theoretically; I haven't tested it fully yet).
 * One includes : This is a small optimization to avoid having the app send a message over the JS bridge to get the initial metadata. We'll see if want to keep it that way. It would be better to get the initial metadata to the app even sooner. I'm open for suggestions here.

.../page/lite/{title}
This route is meant for a new generation lite app; initially targeted for low-powered, older Android devices. The idea is, instead of using a WebView, to use native Android UI components to show the page contents.

This route is currently just using action=mobileview but it's foreseeable that it'll use similar backend calls as the previous route.

More at T90758.

Source
The services are in the following Gerrit repos:
 * 1) mediawiki/services/mobileapps
 * 2) mediawiki/services/mobileapps/deploy

The second repo is for deployment purposes. The first repo contains the implementation of the service routes. Both repos are based on the templates provided by our services team.

Development on local machine
The README.md file in the repo has some great pointers on how to set up and use the service on a dev machine.

Deployment on labs machine
T91794 Deploy experimental version of mobile apps content service

The services are deployed to appservice.wmflabs.org. Here is a simple example for the HTML endpoint: http://appservice.wmflabs.org/en.m.wikipedia.org/v1/mobile/app/page/html/Dilbert

Deployment on Production cluster
T92627 Deployment of Mobile App's service on the SCA cluster

Documentation: Docker + Deployment
 * Note: The deployment preparation requires the use of Docker to make sure the Node modules have the correct binaries. We've been exploring the option of using Boot2docker on Mac OS but ran into issues with the docker script.

TODO: The mobileapps services are still to be deployed to the SCA cluster.