Hi Jeremy,
You bring up several issues here I can address:
- Functional testing by volunteers
- Test case management for persisent, repeatable tests
- Use of beta cluster by multiple teams for multiple purposes
We have tried several experiments enlisting volunteers to test features under development, and determined through experience that such projects cost more than they benefit. All of the software development projects done by WMF are iterative to some extent, meaning that we always have a backlog of features under development. Naive testers, that is, people not following closely the day-to-day activities of these projects, invariable report bugs in three categories and only three categories: WONTFIX; DUPLICATE; and ENHANCEMENT (with the ENHANCEMENT being known issues managed in the product backlog.)
Our needs for persistent test case management is mostly answered by our suites of automated browser tests. However, I do know that our Language team has a desire to manage a suite of test cases for manual testing, which is the sort of thing that Moztrap does. I've discussed this with them, and we intend to use Phabricator to do this. In the very near future, Phabricator will be the issue-tracking system for all of WMF, so it makes sense to use it for test case management also.
Finally, beta cluster has become an important part of our work across virtually every WMF project. The beta labs shared test environment today cannot really properly support all of the projects it is being used for. We've identified the need for at least one additional test environment similar to beta labs, and we are starting work to build such a test environment.
