Wikidata query service/Implementation

Jump to: navigation, search


The source code is in gerrit project wikidata/query/rdf, github mirror:

The GUI source is in wikidata/query/gui project and is the submodule of the main project. The deployment version of the GUI is in production branch, which is cherry-picked from master branch when necessary. Production branch should not contain test & build service files (which currently means some cherry-picks will have to be manually merged).

Labs Deployment (beta)[edit]

Note that currently deployment is via git-fat (see below) which may require some manual steps after checkout. This can be done as follows:

  1. Check out wikidata/query/deploy repository and update gui submodule to current production branch (git submodule update).
  2. Run git-fat pull to instantiate the binaries if necessary.
  3. rsync the files to deploy directory (/srv/wdqs/blazegraph)

See also Wikidata Query service beta.

Production Deployment[edit]

Production deployment is done via git deployment repository wikidata/query/deploy. The procedure is as follows:

  1. mvn package the source repository.
  2. mvn deploy -Pdeploy-archiva in the source repository - this deploys the artifacts to archiva. Note that for this you will need repositories wikimedia.releases and wikimedia.snapshots configured in ~/.m2/settings.xml with archiva username/password.
  3. Install new files (which will be also in dist/target/service-* to deploy repo above. Commit them. Note that since git-fat uses archiva as primary storage, there can be a delay between files being deployed to archiva and them appearing on rsync and ready for git-fat deployment.
  4. Use scap deploy to deploy the new build.

The puppet role that needs to be enabled for the service is role::wdqs.

It is recommended to test deployment checkout on beta (see above) before deploying it in production.

GUI deployment[edit]

GUI deployment files are in repository wikidata/query/deploy-gui branch production. It is a submodule of wikidata/query/deploy which is linked as gui subdirectory.

To build deployment GUI version, use grunt deploy in gui subdir. This will generate patch for deploy repo that needs to be merged in gerrit (currently manually). Then update submodule gui on wikidata/query/deploy to latest production head and commit/push the change. Deploy as per above.


Service wdqs-blazegraph runs the Blazegraph server.

Service wdqs-updater runs the updater. Depends on wdqs-blazegraph.

Maintenance mode[edit]

In order to put the server in the maintenance mode, create file /var/lib/nginx/wdqs/maintenance - this will make all HTTP requests return 503 and the LB will take this server out of rotation. Note that Icinga monitoring will alert about such server being down, so you need to take the measures to prevent it if you are going to do maintenance of the server.

Upgrading customized Blazegraph[edit]

  1. Use script to create Blazegraph binaries. Note that you need to update VERSION in the script. Also note that JAVA_HOME should point to Java 7 home directory (production hosts do not have Java 8 yet).
  2. Use instructions in source repo to upload the new binaries to archiva.
  3. Update Blazegraph version in pom.xml in the source.
  4. Rebuild/redeploy production packages as described above.

Releasing to Maven[edit]

Release procedure described here:

Releasing new version[edit]

  1. Set the version with mvn versions:set -DnewVersion=1.2.3
  2. Commit the patch and merge it (git commit/git review)
  3. Tag the version: git tag 1.2.3
  4. Deploy the files to OSS: mvn clean deploy -Prelease. You will need GPG key configured to sign the release.
  5. Proceed with the release as described in OSS guide.
  6. Set the version back to snapshot: mvn versions:set -DnewVersion=1.2.4-SNAPSHOT
  7. Commit the patch and merge it (git commit/git review)

Updating specific ID[edit]

If there is a need to update specific ID data manually, this can be done using (for ID Q12345): -n wdq -- --ids Q12345,Q6790

The script is located in the root of WDQS deployment directory. Note that each server needs to be updated separately, they do not share databases.


Icinga group

Grafana dashboard:

WDQS dashboard:


If you need more info, talk to User:Smalyshev_(WMF) or anybody from Discovery team.