Continuous integration/Jenkins

Jump to: navigation, search
Jenkins logo with title.svg

Jenkins is a Java tool used to handle recurring tasks such as running tests or building packages. Our primary install is at

The tool is permanently connected to our review tool (Gerrit) and can be made to react on changes submitted to Gerrit. A typical example, is running MediaWiki unit tests whenever a change is submitted to the mediawiki/core.git repository.

The main repository for the layout of Jenkins itself is integration/jenkins.git (installed in /var/lib/jenkins on gallium).

The configuration of individual jobs is abstracted via Jenkins job builder. The jobs are triggered with Zuul (which abstracts Gerrit's events).

You can add new Jenkins slaves.


These services run alongside on the Jenkins server as a result of certain jobs that publish results outside the Jenkins realm for other uses:

Nightly snapshots[edit]

They are being generated at 3am UTC via the Jenkins job "Nightly - MediaWiki core". The ant target is nightly-mediawiki-core and is really straightforward : it get the latest version of master and uses git-archive to generate a zip file. It is then copied under /org/mediawiki/integration/nightly hierarchy which is publicly available via

The latest snapshot is always available at (it is a symbolic link to the latest snapshot).

Android applications[edit]

The WMF produces three Android application (WikipediaMobile, WLMMobile and WiktionaryMobile), hosted on GitHub. A pre commit hook notifies Jenkins whenever a change is submitted in GitHub. The ping back URL is Jenkins will then fetch the branch the commit was made in and build the application.

Jenkins expects an ant script named build.xml in the root directory and will then execute ant debug. Once it completes a build, it copies the generated apk under /srv/org/mediawiki/integration/*Mobile to make it available publicly. For example, the WLMMobile application at .

The mobile team is informed of build completion via an IRC bot idling in their irc channel irc://


Automatic installation[edit]

curl | bash

To make the install go faster, it helps to have a mediawiki-core checkout in ~/src/mediawiki-core - if this repository exists, it will make a local clone. If it doesn't, it will download from gerrit instead (slow!).

Manual installation[edit]


Hung beta code/db update[edit]

This deadlock seems to happen more often than not following or during a database update that is taking a while to complete.

Sometimes you have to do this whole dance several times before Jenkins realizes that the there are a bunch of executors that it can use.

Alternate method:

This second method may interrupt communication between running Jenkins jobs and Zuul but it seems to work even when the offline/online method fails to clear the deadlock.


The Debian init.d script for Jenkins is broken (unable to find the PID, T53817).

Zuul should not be restarted. Zuul preserves the queue and continues after the restart.

Via web interface

Apply the self-serve Jenkins repair!

With a safeRestart any currently running jobs will block a restart until they are canceled. Any long running jobs should be killed. Check for jobs on the main jenkins dashboard, cancel any long-running jobs there. Bonus points: make a note of the patches for which you have canceled jobs on the zuul dashboard, comment "recheck" for any patches in the test queue that you have aborted.

  1. Head to
  2. Login with your labs account being part of the 'wmf' LDAP group
  3. press "Yes"
  4. in #wikimedia-operationsconnect: "!log restarting stuck Jenkins".


On, find the PID of Jenkins (it is a java process) and sudo -u jenkins kill -9 .. it.

Ensure the process is gone (grep through ps aux).

$ sudo /etc/init.d/jenkins start

'demon', 'krinkle', 'reedy', 'mholmquist' have the proper sudo rights. And ops of course :-]


Start Jenkins with Java option:


Text thread dump:

See also[edit]