Setting up private docker registry in AWS and using it with Beanstalk.

Dependence on docker.io index is something of a risk. I've seen sometimes, the pulls for the images located on the remote docker registry timing out. This is one of the few issues, others being large download times for pulling up layers of base image, lack of control on what goes in and who can access it, etc.

Also for large enterprises, ideal solution means having control on what gets stored and what doesn't in the docker registry. Docker public index hardly suits the bill in that regards. But Docker folks are smart, they have designed something called as private registry. You can host it in-house infrastructure or on your favorite cloud account.

Whats fantastic is, you can then pull and push images from this private registry not only locally, but using AWS Beanstalk. This can work wonders both in terms on security and reliability of the registry and its availability, since you control how it runs and who can access images stored. Plus Docker gives an option to use s3 as a storage option, which means the download times for images during deployments should reduce considerably. In the sections that follow, I'll show you how I went about setting a private registry. My setup is shown below -

As you can see, the docker registry will be hosted on an EC2 instance, with the storage set to s3. The beanstalk environment does a docker deployment which know it needs to pull images from the docker registry storage - i.e. s3.

Below are steps getting the registry installed. It assumes you already have provisioned an EC2 instance (m3.large or higher)

Step 1: Install pre-requisites 

The Docker registry is a Python application, so to get it up and running we need to install the Python development utilities and a few libraries:

 $ sudo apt-get -y install build-essential python-dev libevent-dev python-pip liblzma-dev  

Step Two — Install and Configure Docker Registry

To install the latest stable release of the Docker registry, we'll use Python's package management utility pip:

 $ sudo pip install docker-registry  

Now next steps were trial and error for me but you don't have to do this - 

Docker-registry requires a configuration file.

pip by default installs this config file in a rather obscure location, which can differ depending how your system's Python is installed. So, to find the path, we'll attempt to run the registry and let it complain:

 $ gunicorn --access-logfile - --debug -k gevent -b 0.0.0.0:5000 -w 1 docker_registry.wsgi:application

Since the config file isn't in the right place yet it will fail to start and spit out an error message that contains a FileNotFoundError that looks like this:

 FileNotFoundError: Heads-up! File is missing: /usr/local/lib/python2.7/dist-packages/docker_registry/lib/../../config/config.yml

The registry includes a sample config file called config_sample.yml at the same path, so we can use the path it gave us to locate the sample file.

Copy the path from the error message (in this case 

 /usr/local/lib/python2.7/dist-packages/docker_registry/lib/../../config/config.yml 

), and remove the config.yml portion so we can change to that directory:

 $ cd /usr/local/lib/python2.7/dist-packages/docker_registry/lib/../../config/ 

Now copy the config_sample.yml file to config.yml:

 $ sudo cp config_sample.yml config.yml 

The default values in the sample config are fine, so no need to change anything there. Feel free to look through them. If you want to do something more complex like using external storage for your Docker data, this file is the place to set it up. That's outside the scope of this tutorial though, so you'll have to check the docker-registry documentation if you want to go that route.

Now that the config is in the right place let's try to test the server again:

 $ gunicorn --access-logfile - --debug -k gevent -b 0.0.0.0:5000 -w 1 docker_registry.wsgi:application 

You should see output that looks like this:

 2014-10-19 11:12:24 [8641] [INFO] Starting gunicorn 18.0  
 2014-10-19 11:12:24 [8641] [INFO] Listening at: http://0.0.0.0:5000 (8641)  
 2014-10-19 11:12:24 [8641] [INFO] Using worker: gevent  
 2014-10-19 11:12:24 [8641] [INFO] Booting worker with pid: 8643  
 2014-10-19 11:12:24,807 DEBUG: Will return docker-registry.drivers.file.Storage  

Great! Now we have a Docker registry running. Go ahead and kill it with Ctrl+C.

At this point the registry isn't that useful yet — it won't start unless you type in the above gunicorn command. Also, Docker registry doesn't come with any built-in authentication mechanism, so it's insecure and completely open to the public right now.

Step Three - Start Docker Registry as a Service

Let's set the registry to start on system startup by creating an Upstart script.

First let's create a directory for the log files to live in:

 $ sudo mkdir -p /var/log/docker-registry  

Then use your favorite text editor to create an Upstart script:

 $ sudo nano /etc/init/docker-registry.conf

Add the following contents to create the Upstart script:

 description "Docker Registry"  
 start on runlevel [2345]  
 stop on runlevel [016]  
 respawn  
 respawn limit 10 5  
 script  
 exec gunicorn --access-logfile /var/log/docker-registry/access.log --error-logfile /var/log/docker-registry/server.log -k gevent --max-requests 100 --graceful-timeout 3600 -t 3600 -b localhost:5000 -w 8 docker_registry.wsgi:application  
 end script  

If you run:

 $ sudo service docker-registry start  

You should see something like this:

 docker-registry start/running, process 15518 

You can verify that the server is running by taking a look at the server.log file like so:

 tail -100f /var/log/docker-registry/server.log 

If all went well, you'll see something like the output from our previous gunicorn test above.

Now that the server's running in the background, let's move on to configuring Nginx so the registry is secure.

Step Four — Secure Your Docker Registry with Nginx

The first step is to set up authentication so that not just anybody can log into our server.

Let's install Nginx and the apache2-utils package (which allows us to easily create authentication files that Nginx can read).

 $ sudo apt-get -y install nginx apache2-utils 

Now it's time to create our Docker users.

Create the first user as follows:

 $ sudo htpasswd -c /etc/nginx/docker-registry.htpasswd docker-user1 

Create a new password for this user when prompted.

At this point we have a docker-registry.htpasswd file with our users set up, and a Docker registry available. You can take a peek at the file at any point if you want to view your users (and remove users if you want to revoke access).

Next we need to tell Nginx to use that authentication file, and to forward requests to our Docker registry.

Let's create an Nginx configuration file. Create a new docker-registry file, entering your sudo password if needed:

 $ sudo nano /etc/nginx/sites-available/docker-registry 

Add the following content. Comments are in-line.

 # For versions of Nginx > 1.3.9 that include chunked transfer encoding support  
 # Replace with appropriate values where necessary  
 upstream docker-registry {  
  server localhost:5000;  
 }  
 server {  
  listen 8080;  
  server_name my.docker.registry.com;  
  # ssl on;  
  # ssl_certificate /etc/ssl/certs/docker-registry;  
  # ssl_certificate_key /etc/ssl/private/docker-registry;  
  proxy_set_header Host    $http_host;  # required for Docker client sake  
  proxy_set_header X-Real-IP $remote_addr; # pass on real client IP  
  client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads  
  # required to avoid HTTP 411: see Issue #1486 (https://github.com/dotcloud/docker/issues/1486)  
  chunked_transfer_encoding on;  
  location / {  
    # let Nginx know about our auth file  
    auth_basic       "Restricted";  
    auth_basic_user_file  docker-registry.htpasswd;  
    proxy_pass http://docker-registry;  
  }  
  location /_ping {  
    auth_basic off;  
    proxy_pass http://docker-registry;  
  }   
  location /v1/_ping {  
    auth_basic off;  
    proxy_pass http://docker-registry;  
  }  
 }  

And link it up so that Nginx can use it:

 $ sudo ln -s /etc/nginx/sites-available/docker-registry /etc/nginx/sites-enabled/docker-registry  

Then restart Nginx to activate the virtual host configuration:

 $ sudo service nginx restart 

Let's make sure everything worked. Our Nginx server is listening on port 8080, while our original docker-registry server is listening on localhost port 5000.

We can use curl to see if everything is working:

 $ curl localhost:5000 

You should something like the following

 "docker-registry server (dev) (v0.8.1)"

Great, so Docker is running. Now to check if Nginx worked:

This time you'll get back the HTML of an unauthorized message:

 <html>  
 <head><title>401 Authorization Required</title></head>  
 <body bgcolor="white">  
 <center><h1>401 Authorization Required</h1></center>  
 <hr><center>nginx/1.4.6 (Ubuntu)</center>  
 </body>  
 </html>  

It's worthwhile to run these two test commands from a remote machine as well, using the server's IP address instead of localhost, to verify that your ports are set up correctly.

In the Upstart config file we told docker-registry to listen only on localhost, which means it shouldn't be accessible from the outside on port 5000. Nginx, on the other hand, is listening on port 8080 on all interfaces, and should be accessible from the outside. If it isn't then you may need to adjust your firewall permissions.

Good, so authentication is up. Let's try to log in now with one of the usernames you created earlier:

 $ curl USERNAME:PASSWORD@localhost:8080  

If it worked correctly you should now see:

 "docker-registry server (dev) (v0.8.1)"

Step 5: Configure docker-registry with s3 storage.

Edit the config.yml located under docker-registry home. You will observe there are few different environment settings - local, dev, test and prod. I went ahead and configured my prod settings, which look something as below - 

 prod:  
   loglevel: warn  
   storage: s3  
   s3_access_key: _env:AWS_S3_ACCESS_KEY  
   s3_secret_key: _env:AWS_S3_SECRET_KEY  
   s3_bucket: _env:AWS_S3_BUCKET  
   boto_bucket: _env:AWS_S3_BUCKET  
   storage_path: /docker/registry  

Step 6: Add ELB and your instance to it and provide a hostname.

From your AWS account, create an ELB and add your instance to it. As a healthcheck you could provide the / as it should return 200 OK if your setup works as expected.

Step 7: Configure your Dockerrun.aws.json file pointing out the location of image on your private docker registry

 {  
  "AWSEBDockerrunVersion": "1",  
  "Authentication": {  
   "Bucket": "my-bucket",  
   "Key": "mydockercfg"  
  },  
  "Image": {  
   "Name": "gpandit.docker.com/nginx/image-test",  
   "Update": "true"  
  },  
  "Logging": "/var/log/nginx"  
 }  

You can read more about Dockerrun.aws.json for Beanstalk here.

There you have it. If all the setup worked correctly, your Beanstalk should now pull your Docker images from s3 by talking to your private repository. 

Happing Dock(er)ing!