Reading/Web/Projects/NewMobileWebsite/Technical overview

Introduction
This document intends to explain the technical approach to accomplish the requirements for the project outlined in the documentation. See the following sections for links to background information, goals and an overview of the proposal. At the end of the document you can find Frequently asked questions and a Glossary to answer and address common doubts that are not addressed anywhere else.

Background documentation
Project page on mediawiki.org/wiki/Reading/Web/Projects/NewMobileWebsite
 * Proposal
 * High-Level Functional Requirements
 * Roadmap Outline
 * Software projects
 * Phabricator board #marvin
 * Gerrit repository (marvin)

Goals
Goal of the project is to build a mobile website for Wikimedia projects with a focus on better handling poor connectivity scenarios and enabling possible offline reading experiences in the future.

More in detail, derived from the proposal:
 * 1) Optimizing for a modern browsing experience
 * 2) Client side connection management and offline resiliency. Remain functional with good UX and smooth transitions during short bursts of low connectivity or loss of network connection
 * 3) Data efficiency, for low end network connections
 * 4) Providing the ability to implement a fully offline reading experience in the future
 * 5) Inform decisions for other Wikimedia platforms/products in the future. For example:
 * 6) Encourage creation and design of more and better content APIs and services
 * 7) API driven frontends
 * 8) New performance patterns and front-end tooling
 * 9) Possibilities of evolution for the desktop website

Overview
We propose to build an experience that is standalone and fully API based, consuming content from the REST services shared between the different platforms (iOS and Android) and MediaWiki APIs.

The architecture would be in the form of T88301: RFC: Clean front-end - backend separation; make all front-ends API consumers, so both the server and browser clients will rely on shared REST services for content consumption and be in charge of rendering the data. This would allow us to build modern browsing experiences. For example, existing features like Page previews (Desktop), Related articles (Mobile web), and the Feed (Android and iOS native apps), are enabled by such APIs.

It will also allow us to focus on improving data efficiency, by modifying the REST services or creating new layered ones (see Page Content Service Project - closed WIP doc ATM) to focus on transfer size and customize them to just serve the data we need.

In order to enable connection management features for the browser clients and enable having a good architecture to provide a fully offline experience or an installable web app, we need to enrich the browser client application, so that it can perform routing and full rendering without needing a server. That is because in scenarios where a network connection is not available the client needs to be fully capable of navigation, and rendering user saved or client cached data, to provide a capable experience.

At the same time, we need to have an excellent website first load experience, as it is a website that will be navigated to normally from web browsers.

For achieving such isomorphic rendering without duplicating the work, there needs to be a clean separation of responsibilities (routing, rendering, API fetching and state management) in the same language (JS) so that the pieces can be reused in the server and client rendering.

For such reasons, we propose implementing the mobile site as a Progressive Web App, meaning that:
 * The mobile website will be a server side rendered website, to serve very fast and minimal HTML with inlined critical CSS to provide content to the clients as fast as possible.
 * The website will be enhanced as a web application in the client to provide basic connection management features, and...
 * Installing Service Workers where available in capable clients to provide advanced connection management features and (maybe in the future) full offline support and an installable website.

Proposed architecture for the Mobile web PWA

 * Client JavaScript web application
 * UI layer (Router + Views)
 * Application state layer
 * API client layer (REST + MW API)
 * IndexedDB data layer
 * Service worker caching layer
 * Server
 * Node.js server (following Services team recommendations)
 * Reuses from the Client web application for server rendering:
 * UI layer
 * Application state layer
 * API client

Where does this fit in the bigger picture?
See in Page Content Service Project - closed WIP doc ATM), the proposal diagrams. This project would be the Mobile web app box on the diagrams. One of the proposals as an example linked here (not necessarily the chosen/true one):

Initial phase

 * Reading Services
 * Page Content Service project
 * Extra APIs that need development
 * Services team
 * Infrastructure, deployments
 * Design
 * Connection management UX

Next

 * Editing team
 * Visual editing integration

Page rendering, document composition and JSON / HTML APIs in the client
Because of the need to support the most used devices (given a big percentage doesn’t implement Service Workers and those devices probably won’t stop being used for years), right now we need to focus on implementing a solution that gives the best possible experience to both non-service-workers devices (iOS) and service-workers devices (Android). We don’t want to settle on only server-side composition on non-Service Worker devices. Modern iOS devices (for example) are capable and should be able to compose documents on the client too.

As such, initially, we lean towards consuming JSON APIs because JSON parsing is extremely optimized on all browsers, which is very important on mobile devices.

Special attention will be put into rendering the long form content streaming and as fast as possible into the document to be as performant as possible. In non-service-worker capable clients, there are approaches(iframe hack [1 ], streaming ND-JSON [2 ]) for getting content streamed into the window.

Using Service Workers for document composition on server and client
In the future, when most of the devices support Service Worker, we will explore performance optimizations for using HTML APIs instead and extracting metadata and content in an efficient way, streaming and composing documents on the fly using Service Workers both in client and server.

Learnings and use for future improvements to the desktop site
The project focuses on the mobile website given that it’s reduced scope and requirements make it feasible to improve it over a short-medium timespan (1-2 years). The choices and technology will inform future improvements for the desktop site, regarding API driven frontends, use of Service Workers technology, inlining critical CSS, using Parsoid content for reads, and other various topics. Hopefully, we can end up bringing the same sort of improvements adapted for the desktop web to also improve the experience.

What happens with MobileFrontend?
There is no specific plan at the moment, until this project advances and we know what it is good for and if it is successful. Once we know how this project is going, there are many things we can do. I’ll list a few in no particular order (and of course there are many other options):
 * Put it in maintenance mode, no new development will happen
 * Work on aggressively trimming Wikimedia specific parts, and give it to the community as the mobile experience for third party wikis
 * Refactor it to be the grade C experience for old browsers and Opera mini / UC browser removing all the JSÚ
 * Split out a Minerva skin and deprecate support for the MobileFormatter in favor of REST services/upstream changes to core (e.g. section wrapping in the parser)

Root domain and abstraction of language switching
Could we consider making the new website more “app-like” by abstracting away the project domains much like the iOS and Android apps? In this model, the new website would be hosted somewhere generic like: “ https://mobile.wikipedia.org/ ” or “ https://lite.wikipedia.org/ ”. And language switching would happen with a control (much like it does now in the website).

Related: Is a goal to provide a unified experience of all wikimedia projects? Only wikipedias?
Would we provide a single project app e.g. use the app to render content from all Wikimedia projects such as other languages/other projects e.g. wiktionary? This is a topic that needs discussion and exploration as the project evolves. Refer to the parent proposal for discussion.

Which APIs will be used from REST services and which from API.php
General guidelines of “when necessary and when it makes sense” apply. In consultation with the Reading teams (apps and services) and the Services team. Some general principles:
 * High volume read APIs will usually be consumed from the REST layer. Things like page content, page summary, etc.
 * Endpoints that may be needed that don’t exist right now (like page history for example) will have to be created.
 * PHP APIs ready for high volume consumption (like search) we may consume from the existing endpoint and only wrap around a REST service if it is needed and makes sense, and always as a shared effort across the Reading teams so that the API can be used for the different platforms.
 * Non-cacheable APIs like random we’ll consume from API.php.
 * In the specific case of random there is an ongoing effort to create smart random that the apps team were working on, in which case a caching strategy may be worked out and then we’d use the REST service.

How is this going to work with the editing experience?
We have met with the Editing department and discussed different options. Initially in the prototyping phases, we would use the existing mobile web editing experience, which gives the Editing team a stable target to aim at.

If the project advances successfully, when it starts getting out of prototype versions we’ll start collaborating in order to bring the mobile visual editing experience to this project. Given VisualEditor core is a standalone and portable piece of software, and that it has been integrated into different targets (desktop, flow, mobile incoming), ending up integrating it into the specifics of the project as a first class part of it would be the best final outcome and experience for users.

Glossary

 * PWA (Progressive Web App): a term used to denote web apps that uses the latest web technologies. Progressive web apps are technically regular web pages (or websites) but can appear to the user like traditional applications or (native) mobile applications.
 * Varnish: an HTTP accelerator.
 * Node.js: an open-source, cross-platform JavaScript run-time environment for executing JavaScript code server-side.
 * Action: Data representing an intent or an event that happened in the system and that may be of interest to the state and business logic of the application. Example action in mediawiki/extensions/Popups
 * Action creator: Function that given some arguments will perform side effects or trigger actions. Example action creator on mediawiki/extensions/Popups
 * State container: Part of the application that holds all the state of your application. It doesn't let you change that state directly, but instead forces you to describe changes as plain objects called "actions". Actions can be recorded and replayed later, so this makes state management predictable. With the same actions in the same order, you're going to end up in the same state.
 * Change listener: Event listener that is called when the application state changes with the previous and current state. It may then decide to perform side effects or call action creators.

Links

 * Notes from previous discussion with ArchCom 2017/03/15