Wikimedia Labs/Per-project SaltStack remote execution

We'd like to use saltstack for remote execution. Like everything else in labs, we want this to be per-project. Here's a possible implementation for doing so:

Using peer relationships

 * 1) Add a puppet variable, that's completely ignored by puppet called: "salt_peer_master"
 * 2) Use external pillars. Do a search for instances "with salt_peer_master=true". Get the project from that instance's puppet variables and add that to "peered_projects" pillar. Take the instance name, and add that to " _peer_master". Now, do another search for every instance in that project, add the instances to " _peers".
 * 3) Use a jinja template for salt's master configuration file. For each project in "peered_projects", for each instance in " _peer_master" set the commands that are allowed to run for the other peers.

Here's an example configuration:

peer: i-000002.pmtpa.wmflabs: - .*:      minions: i-000000.pmtpa.wmflabs, i-000001.pmtpa.wmflabs

Problems with this solution
The peer relationships don't allow you to specify which peers are allowed to run which commands on which peers. It only allows you to specify which commands a peer can run on all minions. We'd need to modify salt for this. There's an approved feature request for this.