Mostly unsorted late night thoughts right now. Here's what I want:
- If a CI tool spits out a report of errors on specific lines, then we should be able to show those errors inline instead of looking up in the jenkins console the line numbers.
- Multiple tools support auto fixing errors. It should be possible to automatically apply those fixes without manually doing it locally.
- I want to be able to use dependencies defined in a repository when checking out out git repos so dependencies don't have to be defined centrally in CI config (some kind of recursive dependency resolver probably)
- I would like the list of "jobs" to be run for a repo like phan and npm and composer to be defined in the repository itself (like .travis.yml) BUT the job definitions about how to execute "phan" etc. should be defined centrally in CI configuration.
- I would like gate jobs to be run regularly on a cron-ish schedule. Right now extension tests depend upon plenty of inputs that aren't necessarily in that repo, and its common for things to start failing on random patches because those external factors have changed. And if no one submits a patch in a while, it never gets noticed. We could notice it if we ran tests on a cron schedule of some kind, and had a dashboard for failing stuff.
- Smarter tracking of flaky errors, maybe something similar to http://status.openstack.org/elastic-recheck/
- Ability to prototype/debug/change jobs before committing them to version control, as we currently do with jjb.
- Ability to test patches in one repo against all other repositories. Say I make a large/high-risk change to a structure test or class in MediaWiki core, and want to ensure I didn't break anything, or want to gauge the breakage. I want to be able to trigger all extensions to run all of their tests against my MediaWiki core patch. Fedora has a rebuild branch concept that is similar to this IMO.
Maybe some more later. HTH.