Docker Registry as Docker Swarm Service

Une version Française de ce post est disponible.

Registry Configuration

Basics

Usually the /srv/ directory is used to store system-specific data to be served. As such, we'll use /srv/docker/onsen-registry to store all the configuration of our local registry. Every data specific to the registry configuration will be stored on a single docker swarm node and won't be shared with the other nodes of the swarm.

mkdir -p /srv/docker/onsen-registry/

However, as registry data is volatile, we will let the storage location be /var/lib/registry.

mkdir -p /var/lib/registry/

TLS configuration

We don't have a PKI to handle our certificates. So we need to create an X509 certificate and matching private key for our registry. This will enable us to start the registry with TLS enabled and, as a result, use native basic auth. TLS must be configured for basic auth because it sends credentials as clear text.

Once the certificate and key have been generated, we store them as docker swarm secrets on the docker swarm node, so we can use them in the docker swarm service definition.

mkdir -p /srv/docker/onsen-registry/auth
cd /srv/docker/onsen-registry/
openssl req -newkey rsa:4096 -nodes -sha256 -x509 -days 365 -keyout certs/registry.onsen.lan.key -out certs/registry.onsen.lan.crt
docker secret create registry.onsen.lan.key certs/registry.onsen.lan.key
docker secret create registry.onsen.lan.crt certs/registry.onsen.lan.crt

Restricting Access

We generate our basic authentication file using an htpasswd file. The only supported password format is bcrypt.

htpasswd -Bn lenain > auth/htpasswd

Labeling the docker swarm node

We need to add a label to our docker swarm node so that the registry sticks to it. This is needed because the basic auth credentials and registry data will only be reachable from this node filesystems.

docker node update --label-add registry=true kawaii

Registry as Docker Swarm Service

Defining the service with Docker Compose

Here is the docker-compose.yml file :

version: "3.7"
services:
  registry:
    image: "registry:2"
    environment:
      - REGISTRY_AUTH=htpasswd
      - REGISTRY_AUTH_HTPASSWD_REALM="Onsen Registry"
      - REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd
      - REGISTRY_HTTP_ADDR=0.0.0.0:5000
      - REGISTRY_HTTP_TLS_CERTIFICATE=/run/secrets/registry.onsen.lan.crt
      - REGISTRY_HTTP_TLS_KEY=/run/secrets/registry.onsen.lan.key
    secrets:
      - registry.onsen.lan.crt
      - registry.onsen.lan.key
    ports:
      - "5000:5000"
    volumes:
      - type: "bind"
        source: /var/lib/registry
        target: /var/lib/registry
      - type: "bind"
        source: /srv/docker/onsen-registry/auth
        target: /auth
        read_only: true
    deploy:
      placement:
        constraints:
          - node.labels.registry == true
secrets:
  registry.onsen.lan.crt:
    external: true
  registry.onsen.lan.key:
    external: true

Everything is pretty much straitforward :

• The swarm node registry data directory is bind to /var/lib/registry.
• The previously created auth directory is bind as read only so we can set up the authentication scheme as htpasswd.
• The service discovers the X509 certificate and private key secrets from the node, as they are mounted in /run/secrets. TLS is set up using them.
• Port 5000 is exposed as-is on the node.
• Finally, we set a placement constraint so that the registry will only be deployed on a node having the label registry set to true. Obviously, we only want the docker swarm node having the registry data and the registry auth information to have this label.

Deploying the stack

Classic :

docker stack deploy --compose-file docker-compose.yml onsen-registry

Using the registry

Setting up docker daemons

Each docker daemons of the docker swarm nodes needs to trust our newly created registry.

To do this, we need to copy the registry certificate at a specific location on each nodes to let the docker daemons be aware that this registry is trusted.

mkdir -p /etc/docker/certs.d/registry.onsen.lan:5000/
cp registry.onsen.lan.crt /etc/docker/certs.d/registry.onsen.lan:5000/ca.crt

No need to reload the docker daemon.

Check our registry

Setting a credential store

On Debian, we can use pass to store our docker login password.

As root:

apt install pass
curl -sLO https://github.com/docker/docker-credential-helpers/releases/download/v0.6.0/docker-credential-pass-v0.6.0-amd64.tar.gz && \
    tar xvf docker-credential-pass-v0.6.0-amd64.tar.gz && \
    mv docker-credential-pass /usr/bin/. && \
    rm docker-credential-pass-v0.6.0-amd64.tar.gz

As docker user:

gpg2 --gen-key # If necessary
pass init lenain

Log in

Now we can log in our registry from any host of our swarm:

docker login registry.onsen.lan:5000

Pushing and pulling from registry

# Pull from hub
docker pull alpine:latest
# Tag for registry
docker tag alpine:latest registry.onsen.lan:5000/my-alpine
# Push to registry
docker push registry.onsen.lan:5000/my-alpine
# Remove local image
docker rmi registry.onsen.lan:5000/my-alpine:latest
# Pull from registry
docker pull registry.onsen.lan:5000/my-alpine:latest

Log out

If we don't need to use the registry anymore:

docker logout registry.onsen.lan:5000

Checking Registry content

We can query the registry to list all available images. This can be done even if we aren't logged in through docker.

curl -su lenain -k "https://registry.onsen.lan:5000/v2/_catalog"
Enter host password for user 'lenain':
{"repositories":["my-alpine"]}