Wikimedia Apps/Team/RESTBase services for apps/Deployment process

Developer setup for updating deployment repo
Check out the basics for deployment.

We build the deploy repo in Ubuntu with Docker in order to ensure that the Node modules have the correct binaries.

Before you start the actual deploy run through the setup instructions.

Update and build the deployment repo
This assumes you've already run  in the mobileapps repo.

Sync the code and deploy repos with current master:

Optional: Run the tests in Docker:

Optional: Start the service and run the Nagios endpoint checks (see setup instructions here):

Finally, stop the service and build the new commit for the deploy repo:

And push to Gerrit:

You will find the new patch on https://gerrit.wikimedia.org/r/#/projects/mediawiki/services/mobileapps/deploy,dashboards/default:recent.
 * in Gerrit

Troubleshooting
If something goes wrong you'll need to bring the deploy repo to it original state. Commands like these might help: Hint: If the  command fails you can add the   option the next time.

Deploy
The deployment mechanism has been migrated from trebuchet to scap3. See also deployment guide for further info.

In  channel enter: Look at deployment logs: In another terminal start the actual deployment: Consider running following commands from the same directory to check deployment: In case of issues see how to undo deploy.

Once deployment is done go back to your IRC client and enter the following in the   channel: The   is the SHA1 from the source repo. You can find it in the headline of the commit message of the deploy repo, too. Example:

Tagging deployments in Git
Production deployments are tracked with git tags in the main mobileapps repo. The most recent commit included in each deployment is given a tag in this format:  (e.g., ).

The mobileapps repo contains a shell script at scripts/git-deploy.sh that is used to apply these tags. Tags are cryptographically signed and a GPG signing key is therefore required. See the Git tag setup section for the one-time setup of that.

Note: First update the source and deploy repos on your machine if you use another machine for tagging!

Then run:

Example:

Mark tasks as resolved
Mark the tasks in the 'To deploy' column of the MCS sprint board as resolved.

Troubleshooting & Restarting services
Scap3 handles most of the command line steps here. So, this is mostly kept for troubleshooting purposes.

The service is running on the following machines: In your first terminal tail log file: Alternatively: In another terminal restart the mobileapps service Node.js processes: Check version and run the automatic monitoring check manually:

Other things to check:
 * Uptime of service:
 * Versions:
 * If Swagger spec was changed for this deploy:

Wait 5-10 minutes, watching the log file and #wikimedia-operations for alerts. Other things you might want to monitor: See also Dealing with deploy problems and reverting deploys.
 * If only one of the server sees load: (also see each host's weight)
 * Check conftool/eqiad/mobileapps
 * logstash/Kibana
 * ICINGA Web UI
 * ICINGA mobileapps endpoints health: scb1001, scb1002, scb1003, scb1004, scb2001, scb2002, scb2003, scb2004,
 * Ganglia: scb1001, scb1002, scb1003, scb1004, scb2001, scb2002, scb2003, scb2004,
 * Grafana: mobileapps, RESTBase

Links

 * Deployment page on wikitech
 * scap3 documentation
 * How to find basic info of last scap3 deployment
 * older: Marko's deployment page
 * Deployment notes from service template
 * Docker