This page is used to develop possible solutions for bug 53723.
We would like to
- build Debian packages
- still allows operations to review root-level code
- support current deployment methods (git-deploy)
- minimize manual dependency tracking and contrib updates
Deploy repo with code as submodule
Principle: node will search the path for node_modules, so we can also place it in a parent directory.
Deploy repo layout:
- dependencies, updated with
- parsoid js code as submodule
- Debianization and upstart / systemd config submodule. Repo controlled by ops. Configs can also be directly referenced by puppet.
- symlink to parsoid/package.json
- MediaWiki action API configuration, loaded with --config switch to the action API
With this model multiple independent deploy branches (architecture, node version etc) are possible.
Alternative option considered: node_modules as submodule
- Use submodules to manage the debian/, puppet/ and contrib/ files. Operations has merge rights on debian/ and puppet/ so that root-level files can still be reviewed.
- Automatically build debs for third parties from the full repository, either using launchpad or a labs instance.
- Continue to use git-deploy to push out the full repository including contrib/, but deploy the config files using puppet. Submodules are currently disabled for Parsoid, but can be re-enabled when the external cpp submodules are removed.
See also / random notes
- Simple packaging of the source and the upstart / systemd script should be fairly straightforward to debianize no matter what we use. No building needed, 'just' tar up the repository and install the config.
- npm modules can be converted to basic debs with https://npmjs.org/package/npm2debian
- https://github.com/wikimedia-incubator/debianize aims to simplify debian packaging too, but focuses on automake/autoconf. Unclear if this might be useful to us.
- Simple deb packages can be converted to rpms with alien. If we include node_modules, then we should only depend on node and (optionally) logrotate. Node seems to be called 'nodejs' in most distros: 
- http://debdelta.debian.net/ looks like a bandwidth-efficient way to update Debian packages
- mini-dinstall is an alternative to reprepo, which we are currently using for the wmf repos. Reprepo only keeps the very last version of each package, which makes downgrades.. difficult.
- List of node.js / JS packages in Debian
[13:14] <Ryan_Lane> well, we can deploy the code itself using trebuchet [13:14] <Ryan_Lane> puppet can install the upstart and dependencies [13:15] <Ryan_Lane> you don't need to understand salt to use trebuchet ;) [13:15] <Ryan_Lane> https://wikitech.wikimedia.org/wiki/Trebuchet#Adding_a_new_repo [13:16] <Ryan_Lane> also look at how parsoid is configured in puppet: manifests/role/deployment.pp [13:17] <Ryan_Lane> and note that parsoid has a really shitty init script and that instead you can just use service.restart <service> in the checkout_module_calls config
Building debs for deployment
If we wanted to use debs for deployment, we could do the following:
- use a debian repo manager that keeps old versions (mini-dinstall for example)
- set up a deb builder script that can be run from deployers with sudo, somewhat along these lines:
git clone <repourl> cd repo git submodule update --init rm -r debian # throw away unvetted debian dir ln -s <opsrepo>/parsoid/debian . # link in ops-vetted debian dir # add a changes entry with a new version number (automatically incremented) dpkg-buildpackage ... # upload to the repo and return the new version
Then, to deploy:
apt-get install parsoid=<version>, using dsh or salt.
Ideally we'd also deploy to a few canary systems first and (automatically) see that things are alright before upgrading the remaining systems.