Wikimedia Labs/Per-project SaltStack remote execution

We'd like to use saltstack for remote execution. Like everything else in labs, we want this to be per-project. Here's some possible implementations for doing so:

Using peer relationships

 * 1) Add a puppet variable, that's completely ignored by puppet called: "salt_peer_master"
 * 2) Use external pillars. Do a search for instances "with salt_peer_master=true". Get the project from that instance's puppet variables and add that to "peered_projects" pillar. Take the instance name, and add that to " _peer_master". Now, do another search for every instance in that project, add the instances to " _peers".
 * 3) Use a jinja template for salt's master configuration file. For each project in "peered_projects", for each instance in " _peer_master" set the commands that are allowed to run for the other peers.

Here's an example configuration:

peer: i-000002.pmtpa.wmflabs: - .*:      minions: i-000000.pmtpa.wmflabs, i-000001.pmtpa.wmflabs

Problems with this solution
The peer relationships don't allow you to specify which peers are allowed to run which commands on which peers. It only allows you to specify which commands a peer can run on all minions. We'd need to modify salt for this. There's an approved feature request for this.

Use a simple peer relationship a wrapper script and sudo policies
We could set the following peer relationship:

peer: bastion.wmflabs.org: - .*  bastion-restricted.wmflabs.org: - .*

Then, we could write a wrapper script:

salt-project

We'd need to set a grain on the instances based on the project, and have the script make calls based on the grain.

Problems with this solution
With this solution, if the bastion host was compromised, it would mean immediate compromise of all of the instances. Also, it's not easy to make this script fine grained (run this command only on this instance).