User:JHernandez (WMF)/Draft: Invest in the MobileFrontend & MinervaNeue frontend architecture
This draft has been published toː https://www.mediawiki.org/wiki/Reading/Web/Projects/Invest_in_the_MobileFrontend_%26_MinervaNeue_frontend_architecture
The mobile website (specifically MobileFrontend and MinervaNeue codebases) is technologically outdated. This impedes development of new features and maintenance of existing ones. We propose a solution to, and have provided proof of concepts for, iteratively refactoring and revising the existing code in order to achieve the following outputs:
- Enable compressed asset bundling.
- Speed up unit tests and increase code coverage.
- Complete the MinervaNeue migration.
- Review and refactor components - separate presentation and logic.
In addition to solving the stated problems, we expect the completion of these tasks to yield the following outcomes:
- UI feature work is more independent of MediaWiki implementation decisions.
- Greater capacity, and throughput.
- Increased interest, engagement, and collaboration of internal and external contributors.
- Decreased developer onboarding time and enhanced understanding of code.
- Improved data savings and performance.
- Code sharing (pooled features and talent, reduced bugs and maintenance, gained stability).
- Significantly faster and better automated testing, fewer regressions.
- Fewer dependencies on WMF maintained third-party code.
We believe that this initiative is the responsible choice for building on existing work and a stepping stone to future platform evolution.
Given the existing frontend infrastructure and the state of our mobile web frontend code bases, working on frontend (UI features) is challenging and painful in a variety of ways
With the upcoming API focus from the Platform Evolution Program, the rapid innovation by browser vendors (e.g. Service Workers), and the learnings from Extension:Popups (Page previews) and the progressive web app prototypes, we believe we should invest more in our mobile web platform if we want to make future changes to mobile web, adapt to upcoming changes to MediaWiki Core, ease future maintenance, and have a good base for future feature work, whether based on new contributor or reader features.
This project is planned and budgeted in the annual plan, as technical work under the CDP Output 2.7. This document specifies and explains the work and its value, and sets measurable goals to track the progress on the outputs. The project will be worked on by the Readers Web team, with ~2 engineers focusing on this project for continuous amounts of time, leaving other FTEs to work on other goals.
The project is broken up in sub-projects (outputs), that are valuable in isolation, and are part of a step by step plan towards the larger goal. Each sub-project will be tackled sequentially & iteratively and on the production codebases (live websites), to ensure compatibility with the product, avoid divergence and minimise merge conflicts. We will not use big feature branches or require big unveil refactors.
The document follows the annual plan terminology, discussing first Outcomes that we want to accomplish, and then outlining the Outputs (sub-projects) that will be tackled to accomplish those outcomes.
Overview of outcomes
Given our previous work and experience as a team, and the evolution of the industry and global FLOSS community tooling and ecosystem, we believe we can improve the state of the mobile web UI code bases, with the following outcomes:
- Improve developer experience (onboarding, maintenance, development)
- Reduce technical debt and make production improvements
- Separate presentation and logic
- Make Minerva and MobileFrontend better citizens of the MediaWiki ecosystem
See the Outcomes section for more detail.
1. Improve developer experience (on-boarding, maintenance, development)
Enable future use of tooling on the frontend code base to automate regression detection, improve asset sizes, and other productivity enablers.
There are many FLOSS tools, developed by the community and in wide use in the industry, that we can use to help development, maintenance, and preventing regressions.
We will be able toː
- Identify increases in asset sizes before shipping code
- Run our tests more often and faster
- Run our code through tools that make it easier to have cross-browser compatible code reducing overhead and bugs
- Use other tools that reduce bundle sizes in the most aggressive way to get the smallest possible size
- Automatically sort and bundle our frontend assets without manually specifying order, removing race conditions, and making it easier to create and refactor files to make new features or improve existing ones
Speed up development workflows by improving the unit testing process and improve MediaWiki performance in development by decreasing the load on ResourceLoader.
Because of the amount of files and the architecture of the QUnit browser tests, which run in the browser and require a full server roundtrip and ResourceLoader parsing on-demand, the feedback cycle in development for writing and running code and tests is detrimental to properly testing the code and running changes incrementally.
We will enable developers to run isolated unit tests quickly from the command line without having to spin up a server, shortening the feedback cycle and making it easier to write tests and use TDD processes when coding.
We will also remove the load and overhead we put into ResourceLoader by way of having too many modules and files which hurt performance, by bundling and grouping the files with tooling so that the ResourceLoader definitions stay at a minimum and the file order and file bundling into resource modules is automatic and resilient, reducing production bugs and developer overhead and freeing developers to work with and maintain the code more efficiently.
Smoother engineer on-boarding and contributions from the current and next generation of technical volunteers and employees.
Our code will meet expectations of newer developers (hires/volunteers) and be more pleasurable to work with by the team as a whole. This is hard to measure but intuitively valuable and likely to increase throughput of changes.
Currently, we use bespoke code and don’t use efficient tooling. Our code hasn't kept up with the changes through the years. This confuses and slows down the on-boarding process for many of our devs.
Some aspects mentioned above, like on-boarding, are hard to measure. Instead, we plan to measure proxies:
- We will add the following features to the development processesː
- Asset size and performance budget checks for the frontend code are set and running in CI jobs per patch
- Cross browser prefixing and modern Less compiling is set and configured with our supported browsers
- Frontend assets specify their dependencies in the code via standard module systems and are bundled for ResourceLoader to serve
- CLI based QUnit test runner is set up and running in CI jobs per patch
- CLI unit tests can run isolated without having to spin up a server
- We will focus on writing developer documentation that new developers can run through with ease
- Unit tests are migrated to the CLI runner for faster runs and feedback, we will measure the percentage of test files moved to the new test folder and start measuring code coverage and report on it
2. Reduce technical debt and make production improvements
Stop using unmaintained libraries & technologies (like hogan).
Hogan.js is unmaintained and abandoned, and we should stop relying on it when it makes sense and rely on better-supported alternatives. Using abandoned code is a liability for production software, even if it is very well battle tested.
We will improve bytes shipped with minimal effort by relying on tooling.
By using better tooling we can automate many performance improvements that right now are very difficult e.g. tree shaking/better minification, using templates without a runtime payload/compute cost.
Templates will be run in production without parsing/compiling overhead. We will report on the percentage (LOC based) of migrated templates that don't have the runtime overhead.
3. Separate presentation and logic
Decrease the complexity for new features to be developed and reusing existing components by making our code saner to work with.
When the mobile site was built, its component system used an inheritance model with templates and template partials. Projects like React have shown that the alternative of declarative rendering and a component based composition can make development faster. In addition, libraries such as Hogan are no longer maintained, suggesting their days are over.
- We will be able to tie the MobileFrontend code to the style guide and identify missing components.
- We will remove most inheritance chains in classes in favor of composition.
Increase the quality of our code, reducing the numbers of bugs that make it to production.
Many features built by volunteers are not production ready (e.g., Categories) - we will improve/simplify/remove redundant code in the process.
We will provide baselines of test code coverage before and after the refactoring.
Establish clear boundaries from our mobile web frontend and MediaWiki and enable reuse of the mobile web UIs and workflows outside MediaWiki (apps, existing or new products, other projects in the ecosystem, etc).
By looking into the % of code (LOC based) that has been migrated to a library and that can be run without MediaWiki.
4. Make Minerva and MobileFrontend better citizens of the MediaWiki ecosystem
This project will make developers’ lives easier, increase awareness of cross-skin/mobile testing by other teams, and reduce grumbles about the structure and function of MobileFrontend and Minerva in the MediaWiki ecosystem.
Since we split MobileFrontend from Minerva we have seen increased attention and efforts in testing. For instance, some Foundation teams frequently test against the Minerva skin as part of their QA process now (whereas they didn’t before), and there has been renewed interest in Minerva/MobileFrontend. This move helped people conceptualise the different purposes of these 2 repositories. It also guided engineers on where code should live. That said, there is still a little confusion with the setup:
- On the code level with regards to where code should go - where code lives, code review and feature development is a little rougher than it needs to be.
- Editors/third parties have expressed an interest in using Minerva as their mobile AND desktop skin, which is currently a buggy experience. This work will fix those bugs and make Minerva a viable alternative alongside Vector and other skins. It will also improve the overall experience of the Minerva skin.
It will be possible to install Minerva without MobileFrontend (and vice versa). This will be well-documented and the developer community will be notified.
Outputs - Subprojects
Automate asset bundling
The mobile web projects have very extensive manual configuration for the frontend assets and specifying their dependencies. Just the configuration for MobileFrontend has +1000 lines of code and Minerva has ~500 lines of code.
This current state has a negative impact on development practices, ResourceLoader performance, general development, and our ability to change, move and split the code.
In support of
This approach is the same one taken in Extension:Popups (Page Previews), which has proven to be an effective strategy.
Proof of concept
We will go module by module, file by file, migrating the file to the src/ folder like in Extension:Popups, adapting the imports/exports, and moving the QUnit test files to node-qunit (,) if possible/easy.
Progress can be measured by looking at the amount of files and lines of code migrated to the src/ folder.
Speed up unit test execution and increase code coverage
Tests are a crucial part in the development flow to verify that the code works as intended and is future-proof against changes of behavior. With the current stack tests take a long time to run (because of the ResourceLoader strain and the overhead of running through MediaWiki). As a result of the cost of running tests and the slow feedback cycle during the development flow, we haven’t kept up with properly testing the code, and this has negative effects on feature stability and resilience, and on confidence in changing the code.
In support of
Leveraging the Automatic bundling goal, we will follow the Extension:Popups approach and migrate as many unit tests as we easily can to run in the CLI (Node.js based). Tests that make extensive use of MediaWiki or are too tied to ResourceLoader will be considered integration tests and kept in mediawiki-qunit.
Untested files simple enough to test will have new tests added in the process of migration for Automatic bundling.
Proof of concept
Progress can be measured by looking at the amount of files migrated to the src/ folder, like in the previous sub-project (automatic bundling), and the amount of test files moved or left from tests/qunit to tests/node-qunit.
Currently, we have low test coverage and are unable to accurately measure test coverage. Test coverage will be measured using istanbul as files are migrated to node-qunit and we’d aim for ~75% coverage.
Make Minerva independent of MobileFrontend
In support of
3rd party support is important to the MediaWiki community and causes the team extra maintenance effort (dealing with talk page questions and bug reports). Currently there is an implicit relationship where Minerva requires installing and configuring MobileFrontend, and MobileFrontend requires Minerva. As such, using only MobileFrontend (with Vector for example) or using only the skin Minerva (on desktop) is complicated and causes many issues to MediaWiki administrators.
We will make Minerva an independent skin that can easily be used independently, like other skins. We will continue to support mobile-friendly special pages inside MobileFrontend and clarify the purpose of MobileFrontend and the function it serves (which currently causes a lot of confusion).
We will make MobileFrontend use a UI library for the mobile-friendly Special pages provided. Minerva will also use the same mobile-friendly UI library (which could be used by any skins or other projects in the future).
To do so, we will migrate all UI components and assets into a library, which will be stored in the Minerva skin code repository for development convenience, but also published as an independent library.
Going forward, any engineer working on mobile web - Minerva or related features - will work in the Minerva skin repo. MobileFrontend will then effectively live in maintenance mode.
For WMF products, MobileFrontend will use the installed skin Minerva to get the UI library needed.
For 3rd parties who want to use MobileFrontend with another skin that’s not Minerva we will provide an additional install step for them to do so. In exchange for this slight discomfort they will have more control of their user experience. Websites not affiliated with the WMF (e.g. wikitravel) will have more reason not to inherit Wikipedia’s “brand”.
We believe this will reduce the amount of the work the team has to do in the arena of third party support (we get plenty of bug reports), freeing us to focus on products.
Implementation details may change, depending on activity in T107561 (how will MobileFrontend use Minerva’s components/vice versa).
We’ll email wikitech-l and mobile-l to let MediaWiki community members know about this change, before start as a consultation, sending updates via email with the monthly readers email and after finishing presenting the results.
Proof of concept
We will monitor the resources/assets in MobileFrontend and how they move to the MinervaNeue repository, (except the special pages sources or other files MobileFrontend specific ones).
Review and refactor components
In support of
We will re-evaluate what we need/want from this at this point. When the previous projects have been successful, and if time allows, we will generate a concrete proposal for next steps. Some goals for this would be:
- Separation of concerns, separating UI and network and business logic
- Making our code testable
- Using composition instead of inheritance in the UI components
- Declarative UI rendering
Proof of concept
We will define this subproject further as we get closer. It is left open ended right now to allow us the opportunity to choose the approach best-suited to the foundation’s long term objectives and the team’s workload during the FY. In the long-term, we believe these changes will help tackle design debt, reduce technical debt, and simplify the UI code to ease onboarding and development of new features, amongst other things.