Core Platform Team/Initiative/Add API integration tests/Initiative Description

Project Leads
Daniel Kinzler

Current state
In planning

Expected start
April 2019

Summary
Create a suite of end-to-end tests for all modules of MediaWiki’s action API.

Significance and motivation
We need to ensure we don’t break anything while refactoring core code for the Decoupling and HTTP API projects. Unit test coverage of MediaWIki core is insufficient for this purpose. Instead of writing unit tests for code what we plan to dismantle, we should instead ensure that the behavior of the outward facing API remains stable. Since all key functionality of MediaWiki can be accessed via that API, testing that API could cover all critical code paths, assuming tests are written for all relevant actions and combinations of parameters.

Milestones and major tasks

 * Work with Release Engineers to validate technology choice to ensure it can be moved to CI and that there isn’t a preferred existing system.
 * Write a spec for the test runner.
 * Write the test runner (custom, or Selenium?)
 * Develop a way to automate the setup and tear down of a test wiki based on the developer environment (Docker)
 * Identify the first trivial tests.
 * Identify critical cross-module use cases to cover in the first iteration of this project. See stories.
 * Write first tests

Outcome
Ultimately: Increase predictability and safety of deploying code changes

Immediately: Allow confident refactoring by improving test coverage.

Baseline

 * No end-to-end API tests exist.

Target

 * API actions that cause database updates have tests covering all relevant modes of operation (combinations of parameters).
 * End-to-end API tests can easily be run in a local development environment
 * Stretch/later: All API actions have tests covering all relevant modes of operation.
 * Stretch/later: End-to-end API tests are run automatically as part of CI
 * Stretch/later: End-to-end API tests are run periodically against deployment-prep ("beta") projects.

Methodology and rationale
Percentage of modules covered, percentage of parameters covered for each module, number of parameter permutations covered, number of cross-module use cases tested.

Time and resource estimate

 * Specifying test runner: 5 hours over two weeks, plus 5 hours of other people's time for review and discussion.
 * Implementing test runner: 20 hours over two weeks, plus 10 hours of other people's time for review and discussion. May need additional time to decide on technology choices.
 * Creating a docker environment to run tests against: 10 hours over two weeks. May need additional time to learn more about docker.
 * Writing tests for cross-module stories and all actions that modify the database: 100 to 200 hours. Most of this work is trivial, but some of it is rather involved. May be slow going at the beginning, while we figure out the fixtures we need.

Dependencies
None

Collaborators
Core Platform

Release Engineering

Stakeholders
API users

Core code developers

Release Engineering

Open questions

 * What testing framework shall be used for the end-to-end tests?
 * Should we use Selenium?

Phabricator
TBD

Relevant materials, plans and RFCs
TBD