software engineering

Discovering service discovery services

By Adam K Dean on

The problem with service discovery, is that in order to use it to discover services, services must first discover the service discovery service.

Docker assigns IP addresses to containers dynamically, so you cannot guaranteee on knowing where exactly a service will be. You don't want to passing the docker socket through to containers either, because that is a Bad Thing.


So the question is, where do you start? You have to start somewhere, you have to have some sort of anchor point that you can gather the rest of your information from. I'm using Consul for my service discovery, and running it inside a container, so this is where I want to start. I want services to be able to talk to Consul and discover other services, but the start point has to be Consul.

I thought about some sort of DNS so I can resolve consul.local, but the problem only shifts to the DNS server. I thought of a docker lookup service, where I can ask for a name and it gives me the containers 172.17.0.??? IP, but again, that simply shunts the responsibility over to the docker lookup application which is another moving part that can break.


The solution was to create a separate virtual network interface which only the Consul services would use. I could then statically assign an IP to the nodes when they start and those IPs would be accessible by any service on the network.


To test this, I created the interface (see below for a permanent solution):

sudo ifconfig eth0:1 netmask up

Then I ran a hello-world app bound to that IP:

docker run --rm -t -i -p tutum/hello-world

I then navigated to and saw the hello world page. This is good, it means eth0:1 works. I wasn't able to create a docker0:1 interface without issues. As of yet I'm not sure why.

The next step is to launch Consul bound to that IP and then have other services connect to Consul now that the IP is known. For Node.js, I'm using node-consul. Once it knows the Consul server location, it should be able to pull all the relevant services.

To start Consul:

# This is only an example snipped from a larger file

readonly NAME="consul-node1"
readonly HOSTNAME="node1"
readonly NETADDR=""

docker run -d -h $HOSTNAME \
    --name $NAME \
    -p $NETADDR:8400:8400 \
    -p $NETADDR:8500:8500 \
    -p $NETADDR:8600:53/udp \
    progrium/consul -server -bootstrap -ui-dir /ui

Now you can connect to knowing that Consul is there.

A more permanent network interface

For a more permanent network interface, edit /etc/network/interfaces and drop in the following at the bottom:

# Private network for Consul services
auto eth0:1
iface eth0:1 inet static

Then bring up the interface:

$ sudo ifup eth0:1

Always check that the interface came up ok, as ifconfig/ifup/ifdown are very strange beings who do what they want, when they want, usually in a way you don't want.


This is less of a structured post and more of a notes-as-I-go format post. Please don't follow these instructions like they will build you a system. They will not. But what they might do is give you an idea as to how you can work certain things. Most of all, they will serve as a great resource for future me.

UFW, Ubuntu, OpenVPN, and Docker

By Adam K Dean on

Today I've been playing with UFW, Ubuntu, OpenVPN, and Docker. A few other things as well but this post relates to those technologies, mainly UFW, Ubuntu, and Docker.

In case you didn't know, like I didn't know, when you expose ports, Docker thinks a great idea would be to put some rules into iptables to allow the traffic to pass through. I'm sure there is a good reason for this, but you may find that when you enable UFW, traffic still gets through. Now that I understand what is going on, it all makes sense, but getting to that point has taken up my evening.

I followed this tutorial on how To Run OpenVPN in a Docker Container on Ubuntu 14.04 which was a breeze. Really highlighted the strengths of Docker. This setup assigns you an IP from

My next step, now that I had VPN access, was to shut down eth0 on the machine, allowing through only ports 22 & 1194/udp. What I didn't know was that Docker was going to be a royal pain in the arse during this time. It would continue to bypass my rules and make me wonder what the hell was wrong with UFW. Nothing was, it was Docker playing with iptables.

To stop Docker from doing this, you have to start the Docker daemon, or Docker engine I think it's now called, with the --iptables=false flag. To do this, on Ubuntu 14.04, open up /etc/default/docker in your favourite text editor and add the following line:


Save that, and then restart the Docker daemon/engine/server thing:

sudo restart docker

Now, I'm not sure what happens here with existing containers, but I went ahead and deleted them and started fresh. Now when you start new containers, there won't be crazy rules bypassing UFW.

For UFW, I had to add a few rules. I added in ports 22 (SSH) and 1194/udp (VPN). I also added a rule to allow all traffic from docker0:

sudo ufw allow 22
sudo ufw allow 1194/udp
sudo ufw allow in on docker0 to any
sudo ufw enable

Next, I started some containers, and tried to access them. No access... turned on the VPN, tried again, and bingo. I had access. It took a while to get there but I got there. Secured access to my containers.

Now I need a long, hot bath to relax. Thanks a lot Docker!

Get container ID from within container

By Adam K Dean on

To get the container ID from within a container, use this:

$ cat /proc/self/cgroup | grep "docker" | sed s/\\//\\n/g | tail -1

I'll probably stick it in an npm module at some point.

Shell settings for safer scripts

By Adam K Dean on

I was just reading through one of Progrium's scripts when I came across set -eo pipefail at the beginning of a script. Having not seen that before, I decided to Google. This is the result of that.

You can use set to manipulate shell variables and functions. Some of these can help you write safer scripts.

set -e

If any command fails, set -e will make the entire script fail, rather than just skipping onto the next line. If you want to allow a line to fail then you can pop || true onto the end of it.

set -u

This will treat unset variables as an error, and immediately exit the script.

set -o pipefail

By default only the last command in a list of piped commands returns a failure code if it fails. By using set -o pipefile, if any of the commands fail, the line will fail. Using this with set -e means that if any command in a piped command fails, the script will fail.

Now back to my reading...

Install Consul on Ubuntu 14.04

By Adam K Dean on

To install Consul on Ubuntu 14.04, first make sure you have unzip available:

$ apt-get install -y unzip

Now, grab the Consul archive, make sure to get the latest & the right architecture, at the time of writing it is 0.5.0, and for Ubuntu it's linux_amd64:

$ wget

Now unzip it.

$ unzip

Now move consul to somewhere in your PATH:

$ mv consul /usr/bin/local/consul

Finally, check it works:

$ consul --version

Consul v0.5.0
Consul Protocol: 2 (Understands back to: 1)

Good job!