Wikimedia Apps/Team/RESTBase services for apps/Deployment process/Setup notes

Here are the detailed notes to get ready to update the deployment repo for our mobileapps RESTBase service on a Ubuntu box setup (inside a VirtualBox VM). Most also applies for setting up directly on a MacOS machine using Docker for Mac.

Additional Software on Ubuntu

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install curl git nodejs nodejs-legacy npm git-review

Git setup

Basic Gerrit setup

This is more a reminder in case you setup a new machine, as I was doing that initially in a fresh Ubuntu install inside VirtualBox.

See Gerrit/Tutorial#Set_Up_SSH_Keys_in_Gerrit.

Setup source and deploy repos

Note that you have to set an environment variable here for the setup only (temporarily, no need to stick that into an rc script):

GERRIT_USER=
BASE_DIR=~/code/mcs
SRC_DIR=${BASE_DIR}/mobileapps
DEPLOY_DIR=${BASE_DIR}/deploy

mkdir -p ${BASE_DIR}
cd ${BASE_DIR}
git clone ssh://${GERRIT_USER}@gerrit.wikimedia.org:29418/mediawiki/services/mobileapps
git clone --recursive ssh://${GERRIT_USER}@gerrit.wikimedia.org:29418/mediawiki/services/mobileapps/deploy
cd ${DEPLOY_DIR}
git submodule update --init
git review -s
cd ${SRC_DIR}
git review -s
git config deploy.dir ${DEPLOY_DIR}
git config deploy.name mobileapps

Git user name and email

You may also need to set your email and name in git if you haven't done so. See the Gerrit/Tutorial#Set_your_username_and_email.

Git tag setup

Git tags are cryptographically signed and a GPG signing key is therefore required.

gpg --list-keys (or get GPG if you need it)

If no keys are listed or you just installed GPG, create a new signing key with:

gpg --gen-key

Then configure the new key (or desired existing key) as the signing key for the mobileapps repo:

git config user.signingkey <keyId> (add the --global flag if you want to use this signing key for all repos)

For more information straight from the source: https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work

Docker

On MacOS

If running on Mac OS you can install Docker for Mac.

On Ubuntu

The easiest way to install Docker:

curl https://get.docker.com/ > install-docker.sh
sudo bash install-docker.sh

Then, you need to add your user to the docker group, follow the note at the end of the docker commands to do that. In other words:

sudo gpasswd -a $USER docker

After all of that, reboot your VM. Then, you can run the tests inside Docker:

cd ~/code/mcs/mobileapps
./server.js docker-test --verbose

Optional: Nagios Setup

Checking the service with Nagios before building the deploy repo can help to identify problems and prevent deployment reverts.

(Note: These instructions are only the minimum required for command line usage, adapted from the Nagios Ubuntu Quickstart guide.)

Install prereqs:

sudo apt-get install apache2 libapache2-mod-php5 build-essential libgd2-xpm-dev

Create a Nagios user:

sudo -s
cd /usr/sbin
useradd -m -s /bin/bash nagios
passwd nagios  #any password is fine
groupadd nagios
usermod -G nagios nagios
groupadd nagcmd
usermod -a -G nagcmd nagios
usermod -a -G nagcmd www-data
exit

Download, build, and install Nagios:

wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.3.tar.gz
tar xzf nagios-3.2.3.tar.gz
cd nagios-3.2.3
./configure --with-command-group=nagcmd
make all
sudo -s
make install
make install-init
make install-config
make install-commandmode
exit

Download, build, and install nagios-plugin:

wget http://www.nagios-plugins.org/download/nagios-plugins-2.1.1.tar.gz
tar xzf nagios-plugins-1.4.11.tar.gz
cd nagios-plugins-2.1.1
./configure --with-nagios-user=nagios --with-nagios-group=nagios
make
sudo make install

Start Nagios:

sudo -s
ln -s /etc/init.d/nagios /etc/rcS.d/S99nagios
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
/etc/init.d/nagios start
exit

Download the WMF service checker plugin:

wget https://raw.githubusercontent.com/wikimedia/operations-puppet/production/modules/service/files/checker.py

Now, to run the endpoint checks, start the service, then do:

python ./checker.py 127.0.0.1 http://localhost:6927

Once that is all working you can prepare the deploy repo as part of the deployment process.