Hoarde

Wikimedia Foundation projects Hoarde Persistent caching of linked artifacts. Group: SRE/Data Persistence Start: 2025-12-01 Backlog: #cassandra Lead: Eric Evans

Hoarde is a service for the persistent caching of linked artifacts.

API

GET /v1/{name}/{wiki}/{page:[0-9]+}/{revision:[0-9]+}

• {name}: cache name; Artifact namespace identifier
• {wiki}: Wikimedia project identifier
• {page}: MediaWiki page ID (integer)
• {revision}: MediaWiki revision ID (integer)

Retrieves the cached artifact for a specific revision.

Setting the Cache-Control header to no-cache will cause any previously stored output to be disregarded, and the lambda consulted for a new artifact.

Note: Lambdas determine the Content-Type returned.

GET /v1/{name}/{wiki}/{page:[0-9]+}

• {name}: The cache name; The artifact namespace identifier
• {wiki}: Wiki/project identifier
• {page}: MediaWiki page ID (integer)

Retrieves the cached artifact with the highest revision ID. If there are no cached artifacts, an RFC7807 problem object is returned with a 404 status code.

Note: The artifact's revision ID is returned in the X-Revision-Id header.

GET /healthz

Used to established readiness; Returns a JSON-encoded object with version and build info.

GET /metrics

Prometheus metrics

Developing

Setup

$ git clone https://gitlab.wikimedia.org/repos/sre/hoarde.git

Requirements

• git (apt install git)
• Go 1.21+ (apt install golang-go)
• make (apt install build-essential)
• Protobuf compiler (apt install protoc-gen-go protoc-gen-go-grpc)
• goimports (go install golang.org/x/tools/cmd/goimports@latest)

Build

make

Builds the project binaries.

make build-image

Results in a Docker image, hoarde:build, that recreates the environment used to build hoarde (one with build dependencies installed).

make test-image

Results in a Docker image, hoarde:test, useful for running tests in a container (ala docker run -it --rm hoarde:test)

make production-image

Results in a Docker image, hoarde:production that matches what is uploaded to the official repository by CI.

Testing

make check

Runs imports analysis, linting, etc.

make test

Unit tests with mocks.

make functional-test

Functional tests; Requires a running Cassandra cluster.

Releases

To release a new version of Hoarde, create an annotated tag, and push it to GitLab.

$ VERSION="v1.0.0"
$ git tag -am "$VERSION release" $VERSION main
$ git push origin $VERSION

Refer to the semver guidelines in selecting a version

Running

The Wikimedia Foundation runs Hoarde in production using Kubernetes; The easiest way to run the service is to use the Docker image.

Setup

Choosing a Docker image

The Foundation's Docker registry does not implement the latest tag, so browse the list of available image tags and select one appropriate. We'll use v1.3.0 in the following examples.

$ IMAGE_TAG=v1.3.0
$

Configuration

You will need a configuration file for Hoarde. The source repository contains a well-commented sample file, make a copy and edit accordingly.

Cassandra

Cassandra is required, and a keyspace and table(s) must exist. The example below demonstrates creation of a keyspace named artifacts, suitable for a local test environment. Note: Creating a keyspace suitable for production use is outside the scope of this document.

$ cqlsh
Connected to Test Cluster at 127.0.0.1:9042
[cqlsh 6.1.0 | Cassandra 4.1.11-SNAPSHOT | CQL spec 3.4.6 | Native protocol v5]
Use HELP for help.
cqlsh> CREATE KEYSPACE artifacts WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
cqlsh>
$

The source repository contains a schema file that can be used to create tables. You need one table for each cache configured in the tables section of the configuration file.

$ KEYSPACE=artifacts
$ TABLE=topics
$ sed "s/{keyspace_name}/${KEYSPACE}/g; s/{table_name}/${TABLE}/g" schema.cql | cqlsh

Starting a container

The following example creates an ephemeral container (--rm) that runs in the foreground. It assumes your config file (config.yaml) is in the current directory, and that Cassandra is running on the local host (--network=host).

$ docker run -it --rm -v "$(pwd)":/etc/hoarde --network=host \
    docker-registry.wikimedia.org/repos/sre/hoarde:$IMAGE_TAG ./hoarde -config /etc/hoarde/config.yaml
{"@timestamp":"2026-05-13T23:23:20Z","message":"Initializing service linked-artifact-cache (Version: v1.3.0, Go: go1.24.9, Build host: buildkitsandbox, Timestamp: 2026-05-12T14:50:18:UTC)","log":{"level":"INFO"},"service":{"name":"linked-artifact-cache"},"ecs":{"version":"1.11.0"}}
{"@timestamp":"2026-05-13T23:23:20Z","message":"Listening on localhost:8181","log":{"level":"INFO"},"service":{"name":"linked-artifact-cache"},"ecs":{"version":"1.11.0"}}
{"@timestamp":"2026-05-13T23:23:20Z","message":"Cassandra: Opened new connection to 127.0.0.1","log":{"level":"DEBUG"},"service":{"name":"linked-artifact-cache"},"ecs":{"version":"1.11.0"}}
{"@timestamp":"2026-05-13T23:23:20Z","message":"Cassandra: Opened new connection to 127.0.0.1","log":{"level":"DEBUG"},"service":{"name":"linked-artifact-cache"},"ecs":{"version":"1.11.0"}}
{"@timestamp":"2026-05-13T23:23:20Z","message":"Cassandra: Opened new connection to 127.0.0.1","log":{"level":"DEBUG"},"service":{"name":"linked-artifact-cache"},"ecs":{"version":"1.11.0"}}
{"@timestamp":"2026-05-13T23:23:20Z","message":"Cassandra: Opened new connection to 127.0.0.1","log":{"level":"DEBUG"},"service":{"name":"linked-artifact-cache"},"ecs":{"version":"1.11.0"}}
...

See also

• Linked Artifact Cache