Fixing HTTP 404: “invalid character ‘p’ after top-level value” when running a Docker registry behind Traefik reverse proxy

Christopher Klein 2022-12-13

Fixing HTTP 404: “invalid character ‘p’ after top-level value” when running a Docker registry behind Traefik reverse proxy

For ninkik, we experimented with Traefik as a reverse proxy to restrict access to specific Docker image tags. During our tests, we stumbled upon the issue that in the local development environment, a docker push returned the error

error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"

Reproducing the issue in our test setup

Our local development environment had been a pretty simple setup:

a working registry with the official Docker registry image registry:2 without any HTTPS/certificates enabled
a traefik single-binary to make it easier to develop a custom plug-in for ninkik

Our test setup

When doing a simple docker push, the error occurred:

$ docker push localhost:4000/registry
Using default tag: latest
The push refers to repository [localhost:4000/registry]
065fcc6943fa: Preparing
96674a19c926: Preparing
7dd7303d54a9: Preparing
cf1e107ba39e: Preparing
e5e13b0c77cb: Preparing
error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"

Troubleshooting time

Google spit out the following results

Setting REGISTRY_HTTP_RELATIVEURLS
Updating the path prefix
and some other, Harbor-related issues

but none of them fixed the issue in the first place. To make the issue easier to debug, we enabled DEBUG output as JSON in our config.toml:

[log]
    level = "DEBUG"
    filePath = "logs/traefik.log"
    format = "json"

[accessLog]
    filePath = "logs/access.log"
    format = "json"

and started the traefik binary with the option to log the request headers:

./traefik --configfile config.toml --accessLog.fields.defaultMode="keep" --accessLog.fields.headers.defaultMode="keep"

Without using any further strace debugging, the access logs pointed us to the issue:

{"ClientAddr":"127.0.0.1:41136","ClientHost":"127.0.0.1","ClientPort":"41136","ClientUsername":"-","DownstreamContentSize":19,"DownstreamStatus":404,"Duration":24536,"Overhead":24536,"RequestAddr":"localhost:4000","RequestContentSize":0,"RequestCount":1,"RequestHost":"localhost","RequestMethod":"GET","RequestPath":"/v2/","RequestPort":"4000","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"StartLocal":"2022-12-13T11:22:38.728513505+01:00","StartUTC":"2022-12-13T10:22:38.728513505Z","TLSCipher":"TLS_AES_128_GCM_SHA256","TLSVersion":"1.3","level":"info","msg":"","time":"2022-12-13T11:22:38+01:00"}

Due to our local test environment, we had added both registries (the one in the Docker container and our traefik instance) to the insecure-registries field in /etc/docker/daemon.json. But as docker push tries HTTPS at first and ignores any certificate errors if the host is specified in insecure-registries, HTTPS is still used. That process is well documented.

traefik’s configuration had been a simple one and accepted both HTTP and HTTPS traffic on the same port:

[http]
    [http.routers]
        [http.routers.route-to-registry]
            rule="Host(`localhost`)"
            service = "registry-service"
            entryPoints = ["web"]

    [http.services]
        [http.services.registry-service.loadBalancer]
            [[http.services.registry-service.loadBalancer.servers]]
                scheme = "http"
                url = "http://localhost:5000/"

Fixing the issue

To let traefik forward the incoming connection to HTTP, the HTTPS/TLS connection must be terminated. This is done by adding the following empty configuration field to traefik’s configuration file:

[http]
    [http.routers]
        [http.routers.route-to-registry]
            rule="Host(`localhost`)"
            service = "registry-service"
            entryPoints = ["web"]

            [http.routers.route-to-registry.tls]

This is definitely not an approach to fix any situation for this issue, but it might point you in the right direction.

Please note, that it is important to set in http.services.registry-service.loadBalancer.servers the configuration parameters scheme to http and url to http://localhost:5000. Otherwise, TLS termination will not have any effect.

Categories:

Docker