Mobile QA can be broken down into several categories: regression tests, automation tests, crowd sourcing, & device testing. No one solution can solve the needs of the WMF and it takes a blend of all of these into a robust QA plan in order to support the work that the Mobile team is doing.
We have just finished a set of Mobile Browser Regression Tests. We plan to consistently update this document as additional Features are added. We will also add tests around bugs that arise in the future in order to ensure that we are consistently testing the right things. We will also create Application Regression Tests for both Android, iOS and Windows devices.
Automation includes anything that can be done without human intervention. This includes both the concept of unit tests and browser based tests. Automation works best when you know what might break. It works poorly in finding issues that you can't anticipate.
- Jenkins - We already use it and Ant to build the Wikipedia Android app
- Test Swarm - Used by the WMF but not specifically for mobile related projects
Were currently not doing anything within this category. Some potential options include:
- Selenium - Now has automated support for android
- Perfecto/DeviceAnywhere - Has support for writing scripts
Crowd sourcing includes any outreach to real users for debugging our software. It's exceptionally good at testing new user experience, finding problems we can't anticipate, and acts as a great outreach tool. Were currently making use of some crowd sourced options but could be doing even more.
- uTest - Not currently used but would open up our testing user base
- Mailing lists - Actively using mobile-l@
- Twitter - Actively using @WikimediaMobile
- Signpost - Not currently used but could be effective
- m. app (advertising app testing) - Not used
- m. beta (advertising beta mobile web testing) - Beta present but poorly advertised
On device testing is best when trying to reproduce issues. But its slow and cumbersome as you have to either have the real device or work through a remote service.
- Perfecto/DeviceAnywhere - We have an active account with Perfecto
- Real devices - We have a collection of real devices in the office