Continuous integration/Jenkins

Jenkins is a Java tool used to handle recurring tasks such as running tests or building packages. Our primary install is at https://integration.wikimedia.org/ci/.

The tool is permanently connected to our review tool (Gerrit) and can be made to react on changes submitted to Gerrit. A typical example, is running MediaWiki unit tests whenever a change is submitted to the  repository.

The main repository for the layout of Jenkins itself is  (installed in   and is really straightforward : it get the latest version of master and uses   to generate a zip file. It is then copied under   hierarchy which is publicly available via https://integration.wikimedia.org/nightly/.

The latest snapshot is always available at https://integration.wikimedia.org/nightly/mediawiki/core/mediawiki-latest.zip (it is a symbolic link to the latest snapshot).

Wikipedia Android alpha app builds
The mobile apps team uses Jenkins to build its alpha release of the Wikipedia app, available at http://android-builds.wmflabs.org/. Each time a change to /apps/android/wikipedia is merged in Gerrit, an apps-android-wikipedia-publish job is triggered in Jenkins. Upon successful completion, the resulting APK and a JSON file containing the build timestamp are preserved as build outputs.

Every 15 minutes, a script invoked by a cron job running on the (somewhat misleadingly named) 'android-builder' instance in Wikimedia Labs checks for a new APK in the job outputs, and publishes it at the above web site if found.

For more info, see Wikimedia_Apps/Team/Android/App_hacking/Alpha_build_server.

Automatic installation
To make the install go faster, it helps to have a mediawiki-core checkout in ~/src/mediawiki-core - if this repository exists, it will make a local clone. If it doesn't, it will download from gerrit instead (slow!).

Manual installation

 * is the default jenkins configuration directory
 * WM-specific configuration patch I -
 * because some jobs assume jenkins is installed in
 * Download jenkins and place it in ~/.jenkins
 * install the following plugins (download into ):
 * git
 * git-client
 * ansicolor
 * notification
 * scm-api
 * timestamper
 * build-timeout
 * xunit
 * Download and its  ‒ see Continuous_integration/Jenkins job builder for more information. You don't need a password when you install Jenkins locally.
 * WM-specific configuration patch II - Patch the JBB configuration that depends on Zuul (see the mkjenkins script for a diff)
 * If you already have a checkout of mediawiki-core: . Otherwise,
 * Start Jenkins:
 * When Jenkins is running, install the JBB jobs:.
 * When Jenkins is running, install the JBB jobs:.

Hung beta code/db update
This deadlock seems to happen more often than not following or during a database update that is taking a while to complete.


 * Take deployment-tin offline in Jenkins https://integration.wikimedia.org/ci/computer/deployment-tin.eqiad/markOffline
 * Kill any jenkins jobs running on deployment-tin via Jenkins UI
 * Kill all pending jobs in the Jenkins queue that are "waiting on executors"
 * Disconnect deployment-tin https://integration.wikimedia.org/ci/computer/deployment-tin.eqiad/disconnect
 * Bring deployment-bastion back online (button labeled "Bring this node back online")
 * Launch slave agent (there's a button that says this)
 * Check agent log to see that it connected https://integration.wikimedia.org/ci/computer/deployment-tin.eqiad/log

Sometimes you have to do this whole dance several times before Jenkins realizes that the there are a bunch of executors that it can use.

Alternate method:
 * Go to https://integration.wikimedia.org/ci/manage
 * Search page for "Enable Gearman"
 * Un-check the checkbox
 * Save
 * Wait 30s
 * Check the "Enable Gearman" checkbox
 * Save

This second method may interrupt communication between running Jenkins jobs and Zuul but it seems to work even when the offline/online method fails to clear the deadlock.

Restart
Zuul should not be restarted. Zuul preserves the queue and continues after the restart.

Via web interface

Apply the self-serve Jenkins repair!

With a  any currently running jobs will block a restart until they are canceled. Any long running jobs should be killed. Check for jobs on the main jenkins dashboard, cancel any long-running jobs there. Bonus points: make a note of the patches for which you have canceled jobs on the zuul dashboard, comment "recheck" for any patches in the test queue that you have aborted.


 * 1) Head to https://integration.wikimedia.org/ci/safeRestart
 * 2) Login with your labs account being part of the 'wmf' LDAP group
 * 3) press "Yes"
 * 4) in : "!log restarting stuck Jenkins".

Shell

On gallium.wikimedia.org, find the PID of Jenkins (it is a java process) and  it.

Ensure the process is gone (grep through ).

$ sudo /etc/init.d/jenkins start

'demon', 'krinkle', 'reedy', 'mholmquist' have the proper sudo rights. And ops of course :-]

Debugging
Start Jenkins with Java option:

-Dhudson.plugins.git.GitSCM.verbose="true"

Text thread dump: https://integration.wikimedia.org/ci/monitoring?part=threadsDump