Core Platform Team/Initiative/Add API integration tests/Initiative Description

Project Leads
Daniel Kinzler

Current state
In planning

Expected start
April 2019

Summary
Create a suite of end-to-end tests for all modules of MediaWiki’s action API.

Significance and motivation
We need to ensure we don’t break anything while refactoring core code for the Decoupling and HTTP API projects. Unit test coverage of MediaWIki core is insufficient for this purpose. Instead of writing unit tests for code what we plan to dismantle, we should instead ensure that the behavior of the outward facing API remains stable. Since all key functionality of MediaWiki can be accessed via that API, testing that API could cover all critical code paths, assuming tests are written for all relevant types and combinations of parameters.

Milestones and major tasks

 * Write a spec for the test runner
 * Write the test runner (custom, or Selenium?)
 * Develop a way to automate the setup and tear down of a test wiki based on the developer environment (Docker)
 * Identify the first trivial tests
 * Write first tests

Outcome
Architect our code, services and infrastructure for change and sustainability

Baseline

 * TBD

Target

 * TBD

Methodology and rationale
Percentage of modules covered, percentage of parameters covered for each module, number of parameter permutations covered.

Time and resource estimate
TBD

Dependencies
None

Collaborators
Core Platform

Release Engineering

Stakeholders
API users

Core code developers

Release Engineering

Open questions

 * What testing framework shall be used for the end-to-end tests?
 * Should we use Selenium?

Phabricator
TBD

Relevant materials, plans and RFCs
TBD