Wikimedia Apps/Team/RESTBase services for apps/Deployment process

Developer setup for updating deployment repo
We build the deploy repo in Ubuntu with Docker in order to ensure that the Node modules have the correct binaries.

Before you start the actual deploy run through the setup instructions.

Update the deployment repo
This assumes you've already run  in the mobileapps repo.

You will find the new patch on https://gerrit.wikimedia.org/r/#/projects/mediawiki/services/mobileapps/deploy,dashboards/default:recent.
 * in Gerrit

Troubleshooting
If something goes wrong you'll need to bring the deploy repo to it original state. Commands like these might help: Hint: If the  command fails you can add the   option the next time.

Deploy
In  channel enter: Then start the actual deployment: Make sure we have the latest (i.e. the deploy repo patch from earlier step has been merged): Finalize the deployment: In case of issues see how to undo deploy.

Restart services
The service is running on the following machines, and will need to be restarted on each: In your first terminal tail log file: In another terminal restart the mobileapps service Node.js processes: Run the automatic monitoring check manually:

Wait 5-10 minutes, watching the log file and #wikimedia-operations for alerts.

Other things you might want to monitor: Run some tests from the current machine:
 * logstash/Kibana
 * ICINGA Web UI
 * Ganglia: scb1001, scb1002, scb2001, scb2002
 * Grafana: mobileapps, RESTBase

Then repeat with the next machine.

Once you're done with all of them go back to your IRC client and enter the following in the   channel: The   is the SHA1 from the source repo. You'll find it in the commit message of the deploy repo, too.

Tagging deployments in Git
Production deployments are tracked with git tags in the main mobileapps repo. The most recent commit included in each deployment is given a tag in this format:  (e.g., ).

The mobileapps repo contains a shell script at /scripts/git-deploy.sh that is used to apply these tags. Tags are cryptographically signed and a GPG signing key is therefore required.

(or get GPG if you need it)

If no keys are listed or you just installed GPG, create a new signing key with:

Then configure the new key (or desired existing key) as the signing key for the mobileapps repo:

(add the  flag if you want to use this signing key for all repos)

Make sure the deploy directory on the machine you're running from is up to date with the new deployment. For example, you may need to  the latest changes into the deploy repo if running the script from your host machine after generating the deploy repo in a VM.

Finally,

For more information straight from the source: https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work

Links

 * Marko's deployment page
 * Deployment notes from service template
 * Docker