Wikimedia Apps/Team/RESTBase services for apps/Deployment process

Developer setup for updating deployment repo
Check out the basics for deployment.

We build the deploy repo in Ubuntu with Docker in order to ensure that the Node modules have the correct binaries.

Before you start the actual deploy run through the setup instructions.

Update and build the deployment repo
This assumes you've already run  in the mobileapps repo.

Update the code and deploy repos
Sync the code and deploy repos with current master:

Run the tests in Docker:

Run the nagios endpoint checks (see setup instructions here):

Finally, build the new commit for the deploy repo:

And push to Gerrit:

You will find the new patch on https://gerrit.wikimedia.org/r/#/projects/mediawiki/services/mobileapps/deploy,dashboards/default:recent.
 * in Gerrit

Troubleshooting
If something goes wrong you'll need to bring the deploy repo to it original state. Commands like these might help: Hint: If the  command fails you can add the   option the next time.

Deploy
We're in the process of migrating the deployment mechanism from trebuchet to scap3. Soon the descriptions will be changed to incorporate updated instructions from the deployment guide.

In  channel enter: Look at deployment logs: In another terminal start the actual deployment: Consider running following commands from the same directory to check deployment: In case of issues see how to undo deploy.

Once deployment is done go back to your IRC client and enter the following in the   channel: The   is the SHA1 from the source repo. You'll find it in the commit message of the deploy repo, too.

Troubleshooting & Restarting services
Scap3 handles most of the command line steps here. So, this is mostly kept for troubleshooting purposes.

The service is running on the following machines: In your first terminal tail log file: Alternatively: In another terminal restart the mobileapps service Node.js processes: Check version and run the automatic monitoring check manually:

Other things to check:
 * Uptime of service:
 * Versions:
 * If Swagger spec was changed for this deploy:

Wait 5-10 minutes, watching the log file and #wikimedia-operations for alerts.

Other things you might want to monitor: See also Dealing with deploy problems and reverting deploys.
 * logstash/Kibana
 * ICINGA Web UI
 * Ganglia: scb1001, scb1002, scb2001, scb2002
 * Grafana: mobileapps, RESTBase

Setup
Production deployments are tracked with git tags in the main mobileapps repo. The most recent commit included in each deployment is given a tag in this format:  (e.g., ).

The mobileapps repo contains a shell script at /scripts/git-deploy.sh that is used to apply these tags. Tags are cryptographically signed and a GPG signing key is therefore required.

(or get GPG if you need it)

If no keys are listed or you just installed GPG, create a new signing key with:

Then configure the new key (or desired existing key) as the signing key for the mobileapps repo:

(add the  flag if you want to use this signing key for all repos)

For more information straight from the source: https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work

Tag the repo
Note: First update the deploy repo on your machine if you use another machine for tagging!

Then run:

Links

 * Deployment page on wikitech
 * scap3 documentation
 * How to find basic info of last scap3 deployment
 * older: Marko's deployment page
 * Deployment notes from service template
 * Docker