Parsoid/Packaging

From MediaWiki.org
Jump to navigation Jump to search

This page is used to develop possible solutions for bug 53723.

We would like to

  • build Debian packages
  • still allows operations to review root-level code
  • support current deployment methods (git-deploy)
  • minimize manual dependency tracking and contrib updates

Deploy repo with code as submodule[edit]

Status: Decided to go with this option in the team meeting December 11, 2013. Implemented in parsoid and parsoid/deploy repos.

Principle: node will search the path for node_modules, so we can also place it in a parent directory.

Deploy repo layout:

node_modules 
dependencies, updated with npm install
src 
parsoid js code as submodule
debian 
Debianization and upstart / systemd config submodule. Repo controlled by ops. Configs can also be directly referenced by puppet.
package.json 
symlink to parsoid/package.json
localsettings.js 
MediaWiki action API configuration, loaded with --config switch to the action API

With this model multiple independent deploy branches (architecture, node version etc) are possible.

Alternative option considered: node_modules as submodule[edit]

  • Use submodules to manage the debian/, puppet/ and contrib/ files. Operations has merge rights on debian/ and puppet/ so that root-level files can still be reviewed.
  • Automatically build debs for third parties from the full repository, either using launchpad or a labs instance.
  • Continue to use git-deploy to push out the full repository including contrib/, but deploy the config files using puppet. Submodules are currently disabled for Parsoid, but can be re-enabled when the external cpp submodules are removed.

See also / random notes[edit]

  • Simple packaging of the source and the upstart / systemd script should be fairly straightforward to debianize no matter what we use. No building needed, 'just' tar up the repository and install the config.
  • npm modules can be converted to basic debs with https://npmjs.org/package/npm2debian
  • https://github.com/wikimedia-incubator/debianize aims to simplify debian packaging too, but focuses on automake/autoconf. Unclear if this might be useful to us.
  • Simple deb packages can be converted to rpms with alien. If we include node_modules, then we should only depend on node and (optionally) logrotate. Node seems to be called 'nodejs' in most distros: [1]
  • http://debdelta.debian.net/ looks like a bandwidth-efficient way to update Debian packages
  • mini-dinstall is an alternative to reprepo, which we are currently using for the wmf repos. Reprepo only keeps the very last version of each package, which makes downgrades.. difficult.
  • List of node.js / JS packages in Debian

Deployment[edit]

[13:14] <Ryan_Lane> well, we can deploy the code itself using trebuchet
[13:14] <Ryan_Lane> puppet can install the upstart and dependencies
[13:15] <Ryan_Lane> you don't need to understand salt to use trebuchet ;)
[13:15] <Ryan_Lane> https://wikitech.wikimedia.org/wiki/Trebuchet#Adding_a_new_repo
[13:16] <Ryan_Lane> also look at how parsoid is configured in puppet: manifests/role/deployment.pp
[13:17] <Ryan_Lane> and note that parsoid has a really shitty init script and that instead you can just use service.restart <service> in the checkout_module_calls config

Building debs for deployment[edit]

If we wanted to use debs for deployment, we could do the following:

  • use a debian repo manager that keeps old versions (mini-dinstall for example)
  • set up a deb builder script that can be run from deployers with sudo, somewhat along these lines:
git clone <repourl>
cd repo
git submodule update --init
rm -r debian # throw away unvetted debian dir
ln -s <opsrepo>/parsoid/debian .  # link in ops-vetted debian dir
# add a changes entry with a new version number (automatically incremented)
dpkg-buildpackage ...
# upload to the repo and return the new version

Then, to deploy: apt-get install parsoid=<version>, using dsh or salt.

Ideally we'd also deploy to a few canary systems first and (automatically) see that things are alright before upgrading the remaining systems.