Wikimedia Apps/Team/RESTBase services for apps/Deployment process

Developer setup for updating deployment repo
Check out the basics for deployment.

We build the deploy repo in Ubuntu with Docker in order to ensure that the Node modules have the correct binaries.

Before you start the actual deploy run through the setup instructions.

Update and build the deployment repo
This assumes you've already run  in the mobileapps repo.

Sync the code and deploy repos with current master:

Run the tests in Docker:

Start the service and run the Nagios endpoint checks (see setup instructions here):

Finally, stop the service and build the new commit for the deploy repo:

And push to Gerrit:

You will find the new patch on https://gerrit.wikimedia.org/r/#/projects/mediawiki/services/mobileapps/deploy,dashboards/default:recent.
 * in Gerrit

Troubleshooting
If something goes wrong you'll need to bring the deploy repo to it original state. Commands like these might help: Hint: If the  command fails you can add the   option the next time.

Deploy
We're in the process of migrating the deployment mechanism from trebuchet to scap3. Soon the descriptions will be changed to incorporate updated instructions from the deployment guide.

In  channel enter: Look at deployment logs: In another terminal start the actual deployment: Consider running following commands from the same directory to check deployment: In case of issues see how to undo deploy.

Once deployment is done go back to your IRC client and enter the following in the   channel: The   is the SHA1 from the source repo. You'll find it in the commit message of the deploy repo, too.

Tagging deployments in Git
Production deployments are tracked with git tags in the main mobileapps repo. The most recent commit included in each deployment is given a tag in this format:  (e.g., ).

The mobileapps repo contains a shell script at scripts/git-deploy.sh that is used to apply these tags. Tags are cryptographically signed and a GPG signing key is therefore required. See the Git tag setup section for the one-time setup of that.

Note: First update the source and deploy repos on your machine if you use another machine for tagging!

Then run:

Example:

Troubleshooting & Restarting services
Scap3 handles most of the command line steps here. So, this is mostly kept for troubleshooting purposes.

The service is running on the following machines: In your first terminal tail log file: Alternatively: In another terminal restart the mobileapps service Node.js processes: Check version and run the automatic monitoring check manually:

Other things to check:
 * Uptime of service:
 * Versions:
 * If Swagger spec was changed for this deploy:

Wait 5-10 minutes, watching the log file and #wikimedia-operations for alerts.

Other things you might want to monitor: See also Dealing with deploy problems and reverting deploys.
 * logstash/Kibana
 * ICINGA Web UI
 * Ganglia: scb1001, scb1002, scb2001, scb2002
 * Grafana: mobileapps, RESTBase

Links

 * Deployment page on wikitech
 * scap3 documentation
 * How to find basic info of last scap3 deployment
 * older: Marko's deployment page
 * Deployment notes from service template
 * Docker