Extension:Wikispeech/Build Docker images

How to build and publish Docker images

When needed, we can create new images for our components, such as our language engines (Piper, Matcha etc), wikispeech_server or other components. This can be done following the GitLab Workflows.

To better understand our specific workflows, here is some guidance in complement to the GitLab documentation.

Prerequisites

We use Blubber, a MediaWiki BuildKit tool to define everything needed, making it more human readable, that internally converts to a Dockerfile. Therefore, it is recommended to read the Blubber documentation before starting.

1. After creating a (blank) project, clone it locally and create ./pipeline/blubber.yaml in the project root
2. Most commonly used declarations and structure for our components look like:

   # syntax = docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v1.7.0
   version: v4
   runs:
     insecurely: true # To be able to run as root user
     environment:
       PATH: $PATH:<any extra paths to executables>
   variants:
     setup:
       base: <docker-image-base>
       apt:
         packages:
           <apt-packages>
       builder:
         script:
           <commands>
     test:
       includes: [ setup ]
       copies: [ local ]
       entrypoint: [ ./test.sh ]
     run:
       includes: [ setup ]
       copies: [ local ]
       entrypoint: [ ./run.sh ]
   

3. Build the container:
   docker build -f .pipeline/blubber.yaml --tag <container-name> --target run .
4. Expose ports and run the named container. Example:
   docker run --expose 8787 -p 8787:8787 <container-name>:latest
5. Configure the GitLab CI pipeline, creating a .gitlab-ci.yml file in the project root. Ensure that:
   1. BUILD_VARIANT for test jobs uses test
   2. BUILD_VARIANT for publish jobs uses run

   from the blubber.yaml file

6. Publish image to production:
   1. Add the project to trusted runners and create a merge request
   2. Configure protected tags. Our wildcard is typically test-*
   3. Add and commit the changes and push to origin main
7. When the project is added to trusted runners, and the pipelines have passed, create a new tag:
   1. Example: test-<commit-hash>
8. After a while, the image can be found at Wikimedia docker registry