docker-registry behind a few proxies

Running a private docker-registry behind a few proxies took me while to configure, because I had several things that I couldn’t move. In particularly, it is an nginx in front of everything, and the docker-registry that I wanted as a “real” service, because I am still learning the docker ways, and I don’t want it as a container, yet.

I installed the docker-registry in a KVM VM, on a CentOS 7 – a standard business requirement one might say.

That part is easy: fetch the virt7-testing repo, as described here, and move on:

[virt7-testing]
name=virt7-testing
baseurl=http://cbs.centos.org/repos/virt7-testing/x86_64/os/
enabled=1
gpgcheck=0
exclude=kernel

and fetch those thingies you’ll need, including the latest docker:

yum install docker –disablerepo=extras && yum install docker-registry.x86_64 python-sqlalchemy.x86_64

note that the docker-registry is in extras, thus a two-step installation.

I plan to use Apache with LDAP backend to authenticate docker users:

yum install httpd mod_ssl mod_ldap

Add your users in a group that can use Docker, and state that in Apache’s conf for the virtual server:

<VirtualHost *:443>
ServerName darkhat.simulakrum.org
ServerAlias docker-registry.simulakrum.org

SSLEngine on
SSLCertificateFile /etc/ssl/darkhat/server.crt
SSLCertificateKeyFile /etc/ssl/darkhat/server.key

Header set Host “darkhat.simulakrum.org”
RequestHeader set X-Forwarded-Proto “https”

ProxyRequests off
ProxyPreserveHost on

ProxyPass / http://192.168.122.20:5000/
ProxyPassReverse / http://192.168.122.20:5000/

ErrorLog /var/log/httpd/registry-error.log
LogLevel warn
CustomLog /var/log/httpd/registry-access.log combined

<Location />
Order deny,allow
Allow from all

Options FollowSymLinks
AuthBasicProvider ldap
AuthType Basic
AuthName “Simulakrum OpenLDAP”
AuthLDAPURL “ldaps://192.168.38.11:636/ou=accounts,dc=simulakrum,dc=org?cn?sub?(objectClass=*)”
AuthLDAPBindDN “cn=thebinder,ou=accounts,dc=simulakrum,dc=org”
AuthUserFile /dev/null
AuthLDAPBindPassword XXXXXXXXXX
Require ldap-group cn=Containers,ou=bands,dc=simulakrum,dc=org
</Location>

# Allow ping and users to run unauthenticated.
<Location /v1/_ping>
Satisfy any
Allow from all
</Location>

# Allow ping and users to run unauthenticated.
<Location /_ping>
Satisfy any
Allow from all
</Location>

</VirtualHost>

 

For Nginx, that is in front of everything, add for things for the upstream, too:

## darkhat.simulakrum.org
server {
listen 192.168.2.14:80;
server_name darkhat.simulakrum.org docker-registry.simulakrum.org;
rewrite ^ https://$server_name$request_uri? permanent;
}

server {
listen 192.168.2.14:443;
server_name darkhat.simulakrum.org docker-registry.simulakrum.org;
location / {
proxy_pass https://darkhatssl;
client_max_body_size 0;
chunked_transfer_encoding on;
proxy_set_header X-Forwarded-Proto “https”;
proxy_set_header X-Forwarded-Protocol “https”;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

Apparently, Docker can create problems if there are no max body size and chunked directives.

Next, bind this docker-registry for an IP in the startup-script:

[root@darkhat ~]# grep ExecStart /etc/systemd/system/multi-user.target.wants/docker-registry.service
ExecStart=/usr/bin/gunicorn –access-logfile – –debug –max-requests 100 –graceful-timeout 3600 -t 3600 -k gevent -b 192.168.122.20:5000 -w $GUNICORN_WORKERS docker_registry.wsgi:application –access-logfile /var/log/docker-registry/access.log –error-logfile /var/log/docker-registry/server.log

Fill your docker-registry.yml, making a few changes there too:

[root@darkhat ~]# cat /etc/docker-registry.yml
# All other flavors inherit the `common’ config snippet
common: &common
issue: ‘”docker-registry server simulakrum.org”‘
# Default log level is info
loglevel: _env:LOGLEVEL:info
# Enable debugging (additional informations in the output of the _ping endpoint)
debug: _env:DEBUG:false
# By default, the registry acts standalone (eg: doesn’t query the index)
standalone: _env:STANDALONE:true
# The default endpoint to use (if NOT standalone) is index.docker.io
#index_endpoint: _env:INDEX_ENDPOINT:https://index.docker.io
index_endpoint: _env:INDEX_ENDPOINT:https://darkhat.simulakrum.org
# Storage redirect is disabled
storage_redirect: _env:STORAGE_REDIRECT
# Token auth is enabled (if NOT standalone)
disable_token_auth: _env:DISABLE_TOKEN_AUTH
# No priv key
privileged_key: _env:PRIVILEGED_KEY
# No search backend
search_backend: _env:SEARCH_BACKEND:sqlalchemy
# SQLite search backend
#sqlalchemy_index_database: _env:sqlite:////tmp/docker-registry.db
sqlalchemy_index_database: _env:SQLALCHEMY_INDEX_DATABASE:sqlite:////tmp/docker-registry.db

That’s it for the server part. Restart all services and hop for the best. Now the part that really took me some serious time to figure it out – the clients!

At first, none of the clients I tried was able to log in.  I first made sure that LDAP login is working as it should – nginx accepts the call, transfers it to HTTPS port immediately, and calls Apache there on LAN. First I dropped a separate cert for Apache, and I used the *.simulakrum.org catch-all cert, to reduce the complexity. That proved a good move, because I was immediately able to log into LDAP at Apache’s, and since the docker-registry showed no problems with accessibility, I could have moved to the problem of searching and logging.

The dreadful “–insecure-registry” switch proved useless in every possible combination, with certs in /etc/docker/certs.d/server and other places, combinations, and forms, although it was mentioned in almost every single PR, thread and issue I read about . I was getting a million of those

FATA[0010] Error response from daemon: Server Error: Post https://darkhat.simulakrum.org/v1/users/: x509: certificate signed by unknown authority

whatever I’d do. Now, this works for sure – fetch the cert with

echo -n | openssl s_client -connect darkhat.simulakrum.org:443 -showcerts | sed -n -e ‘/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p’

, and put it in

/usr/share/pki/ca-trust-source/anchors/

for RedHat/Fedora like clients, or in

/usr/local/share/ca-certificates/

for Debian/Ubuntu, or, for OpenSuSE, it worked in

/etc/pki/trust/anchors/

already. Do the update-ca-certificates/update-ca-trust extract afterwards, and restart docker daemon, and you should be able to start using registry, with proper logging.

Last, but not the least – if the searching and/or logging are not working, e.g:

FATA[0008] Error response from daemon: (Code: 404; Headers: map[Content-Length:[233] Host:[darkhat.simulakrum.org] Date:[Sat, 02 May 2015 19:39:43 GMT] Server:[gunicorn/18.0] Content-Type:[text/html; charset=UTF-8]])

or

FATA[0005] Error response from daemon: (Code: 404; Headers: map[Server:[nginx/1.6.0] Date:[Sat, 02 May 2015 19:33:34 GMT] Content-Type:[text/html; charset=UTF-8] Content-Length:[233] Connection:[keep-alive] Host:[darkhat.simulakrum.org]])

look at those settings in docker-registry.yml file up here, especially the standalone, search_backend, sqlalchemy_index_databaes and the index_endpoint lines, and adjust accordingly.

Voila – you now have an old-fashioned old-school non-containerized docker-registry working. Now, you can start learning how to forget about all of this mess, and use the very docker to run docker-registry from within a container.. in a docker… registry.. in a… docker, I guess? 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *