Wikimedia Release Engineering Team/Train development environment/2020-05-14

Links: https://blog.liw.fi/posts/2020/04/09/contractor_build_software_more_securely/

Lars: Maintaining scap -- wanted to run its test suite -- uses tools that are not in debian -- created a solution that is "lovecraftian" -- want a place that's clean of unrelated junk that is easy to recreate and that I can make changes to the environment easily. Experiements: move to python3, etc. Environment where I have root and where I know how to do things. Local machine that I control so I can securely have an SSH key there (I don't use agent forwarding). I did not want to do things like developing scap on the deployment server (even though I have root there). These are the things I wanted and if I can have these other people should as well. If we have this then we might want to use this for other things aside from scap and scap development. We can use that environment for experimenting with new ways of deploying and training people.

Mukunda: affirm

Lars: I played with docker-compose and vagrant and I can't say I'm a fan: I understand why they appeal to people. I have a hobby project called "contractor"

Mukunda: I think we need a vm image (not containers) Jeena: same Mukunda Modell: maybe containers for the deployment targets

Lars: Contractor to build CI builds locally (*does demo*)
 * YAML config file that specifies image, packages to install, build instructions, etc.
 * Spins up VM, etc.

Tyler: Are you proposing this as the solution for the local environment for scap?
 * Lars: Not quite. I think we should use virtual machines.  The interesting part about contractor is it uses nested VMs.  I think we should build on that.  It's supported by Intel CPUs for years now.  Requires a lot of RAM and disk space.

Mukunda: I don't think we really need a production environment. Recreating the things that matter isn't trivial... Never knew you could run VMs inside VMs, so that's cool. I think our previous scap vagrant was almost adequate but not quite. Haven't had complete success with running contractor, but seems tractable.

Brennen: I also tried contractor, it seems like a cool thing. What does the nested vms buy us? More controllable environment Mukunda: Design the tool around the vm hosting tool that you're using -- different abstraction layers for VM hypervisors Lars: In theory I can use the VM that I have in VBox Jeena: If everything's in that one vm, then everything should just work? Lars: Yes

Tyler: how does code get shared? Mukunda/Lars: Scp/rsync Brennen: it's a rabbit hole dealing with nfs mounts

Lars: disk image is copied from outer VM to innerVM -- prebuilt images that are copied over Tyler: how are you building images? Are you installing dependencies as part of run?

Lars: [vmdb2 demo]

Jeena: Why are package dependencies in 2 places? Lars: In case I forget them.

Brennen: Image building is decoupled from contractor -- could we use puppet to build images like this? Lars: the only thing that contractor cares about is running commands over ssh

Tyler: One thing that stands out to me here is building tooling vs. building environments. Maintenance and build of this environment will be costly - is there an off-the-shelf alternative that would give us 90% of this so we don't have to spend time on tooling? Lars: I don't know. I build this stuff for myself so I use it... Mukunda: Have seen some things, but they don't seem very Debian-friendly.
 * Ubuntu has multipass: https://multipass.run/

Lars: should we have this environment Mukunda: Would've been nice to have the environment a long time ago Brennen: it seems clear that to make changes to scap we need an environment; the nested vms have grown on me Jeena: I think we need an environment, I think nested vms is fine -- may simplify things for use -- I was thinking about dependencies we would install -- use puppet to build vms -- I'm not sure what contractor adds Mukunda: nested VMs Brennen: As far as time spent developing tooling goes -- it seems like this could be a fairly thin layer -- this is not a general purpose tool

(Some back and forth around off-the-shelf vs. Contractor or a bespoke tool we build etc.) https://www.packer.io/

Lars: How is everyone running VMs?

Mukunda: I've used puppet apply

Tyler: This is what Antoine's thing did - debootstrap, ops/puppet, built a qcow2 image

vmdb2: * https://vmdb2.liw.fi/ * https://gitlab.com/larswirzenius/vmdb2/

Contractor is for managing nested VMs

TODO: Lars to create image for other to try TODO: Jeena to demo packer TODO: thcipriani to set next meeting TODO: Mukunda investigate cloud-init