Wikimedia Apps/Team/RESTBase services for apps/Deployment process

Developer setup for updating deployment repo
Check out the basics for deployment.

We build the deploy repo in Ubuntu with Docker in order to ensure that the Node modules have the correct binaries. We have run into issues with unit tests failing when running Docker for Mac, and therefore recommend building the deploy repo in Docker running on a Linux host.

Before you start the actual deploy run through the setup instructions.

Update and build the deployment repo
This assumes you've already run  in the mobileapps repo.

Sync the code and deploy repos with current master:

Run the tests in Docker:

Start the service and run the Nagios endpoint checks (see setup instructions here):

Finally, stop the service and build the new commit for the deploy repo:

And push to Gerrit:

You will find the new patch on https://gerrit.wikimedia.org/r/#/projects/mediawiki/services/mobileapps/deploy,dashboards/default:recent.
 * in Gerrit

Troubleshooting
If something goes wrong you'll need to bring the deploy repo to it original state. Commands like these might help: Hint: If the  command fails you can add the   option the next time.

Deploy
The deployment mechanism has been migrated from trebuchet to scap3. See also deployment guide for further info.

Check out  on IRC to make sure there's nothing blocking the deploy.

Look at deployment logs: In another terminal start the actual deployment: The command will deploy first on the canary server scb2001.codfw.wmnet. In a different terminal you can log in to the canary server and verify that the service responds to expectations. Examples: Once satisfied press  in the deployment terminal to continue deploying on the other servers.

The string parameter for the scap deploy command will show up in IRC  and SAL. Once for the start and then at the end.

Consider running following commands from the same directory to check deployment: In case of issues see how to undo deploy.

Tagging deployments in Git
Production deployments are tracked with git tags in the main mobileapps repo. The most recent commit included in each deployment is given a tag in this format:  (e.g., ).

The mobileapps repo contains a shell script at scripts/git-deploy.sh that is used to apply these tags. Tags are cryptographically signed and a GPG signing key is therefore required. See the Git tag setup section for the one-time setup of that.

Note: First update the source and deploy repos on your machine if you use another machine for tagging!

Then run:

Example:

Mark tasks as resolved
Mark the tasks in the 'To deploy' column of the MCS sprint board as resolved.

Troubleshooting & Restarting services
Scap3 handles most of the command line steps here. So, this is mostly kept for troubleshooting purposes.

The service is running on the following machines: In your first terminal tail log file: Alternatively: In another terminal restart the mobileapps service Node.js processes: Check version and run the automatic monitoring check manually:

Other things to check:
 * Uptime of service:
 * Versions:
 * If Swagger spec was changed for this deploy:

Wait 5-10 minutes, watching the log file and #wikimedia-operations for alerts. Other things you might want to monitor: See also Dealing with deploy problems and reverting deploys.
 * If only one of the server sees load: (also see each host's weight)
 * Check conftool/eqiad/mobileapps
 * logstash/Kibana
 * ICINGA Web UI
 * ICINGA mobileapps endpoints health: scb1001, scb1002, scb1003, scb1004, scb2001, scb2002, scb2003, scb2004,
 * Ganglia: scb1001, scb1002, scb1003, scb1004, scb2001, scb2002, scb2003, scb2004,
 * Grafana: mobileapps, RESTBase

Links

 * Deployment page on wikitech
 * scap3 documentation
 * How to find basic info of last scap3 deployment
 * older: Marko's deployment page
 * Deployment notes from service template
 * Docker