Parsoid/Packaging

This page is used to develop possible solutions for bug 53723.

We would like to


 * build Debian packages
 * still allows operations to review root-level code
 * support current deployment methods (git-deploy)
 * minimize manual dependency tracking and contrib updates

Plan: deploy repo with code as submodule
Status: Decided to go with this option in the team meeting December 11, 2013. Implemented in parsoid and parsoid/deploy repos.

Principle: node will search the path for node_modules, so we can also place it in a parent directory.

Deploy repo layout:
 * node_modules : dependencies, updated with
 * src : parsoid js code as submodule
 * debian : Debianization and upstart / systemd config submodule. Repo controlled by ops. Configs can also be directly referenced by puppet.
 * package.json : symlink to parsoid/package.json
 * localsettings.js : web api configuration, loaded with --config switch to the web api

With this model multiple independent deploy branches (architecture, node version etc) are possible.

Alternative option considered: node_modules as submodule

 * Use submodules to manage the debian/, puppet/ and contrib/ files. Operations has merge rights on debian/ and puppet/ so that root-level files can still be reviewed.
 * Automatically build debs for third parties from the full repository, either using launchpad or a labs instance.
 * Continue to use git-deploy to push out the full repository including contrib/, but deploy the config files using puppet. Submodules are currently disabled for Parsoid, but can be re-enabled when the external cpp submodules are removed.

See also / random notes

 * Simple packaging of the source and the upstart / systemd script should be fairly straightforward to debianize no matter what we use. No building needed, 'just' tar up the repository and install the config.
 * npm modules can be converted to basic debs with https://npmjs.org/package/npm2debian
 * does not work so well in my testing though Gabriel Wicke (GWicke) (talk) 23:23, 16 December 2013 (UTC)
 * https://github.com/wikimedia-incubator/debianize aims to simplify debian packaging too, but focuses on automake/autoconf. Unclear if this might be useful to us.
 * Simple deb packages can be converted to rpms with alien. If we include node_modules, then we should only depend on node and (optionally) logrotate. Node seems to be called 'nodejs' in most distros:
 * http://debdelta.debian.net/ looks like a bandwidth-efficient way to update Debian packages
 * mini-dinstall is an alternative to reprepo, which we are currently using for the wmf repos. Reprepo only keeps the very last version of each package, which makes downgrades.. difficult.

Deployment
[13:14]  well, we can deploy the code itself using trebuchet [13:14]  puppet can install the upstart and dependencies [13:15]  you don't need to understand salt to use trebuchet ;) [13:15]  https://wikitech.wikimedia.org/wiki/Trebuchet#Adding_a_new_repo [13:16]  also look at how parsoid is configured in puppet: manifests/role/deployment.pp [13:17]  and note that parsoid has a really shitty init script and that instead you can just use service.restart in the checkout_module_calls config

Building debs for deployment
If we wanted to use debs for deployment, we could do the following:

git clone cd repo git submodule update --init rm -r debian # throw away unvetted debian dir ln -s /parsoid/debian. # link in ops-vetted debian dir dpkg-buildpackage ...
 * use a debian repo manager that keeps old versions (mini-dinstall for example)
 * set up a deb builder script that can be run from deployers with sudo, somewhat along these lines:
 * 1) add a changes entry with a new version number (automatically incremented)
 * 1) upload to the repo and return the new version

Then, to deploy:, using dsh or salt.

Ideally we'd also deploy to a few canary systems first and (automatically) see that things are alright before upgrading the remaining systems.