The problem with service discovery, is that in order to use it to discover services, services must first discover the service discovery service.
Docker assigns IP addresses to containers dynamically, so you cannot guaranteee on knowing where exactly a service will be. You don't want to passing the docker socket through to containers either, because that is a Bad Thing.
So the question is, where do you start? You have to start somewhere, you have to have some sort of anchor point that you can gather the rest of your information from. I'm using Consul for my service discovery, and running it inside a container, so this is where I want to start. I want services to be able to talk to Consul and discover other services, but the start point has to be Consul.
I thought about some sort of DNS so I can resolve
consul.local, but the problem only shifts to the DNS server. I thought of a docker lookup service, where I can ask for a name and it gives me the containers
172.17.0.??? IP, but again, that simply shunts the responsibility over to the docker lookup application which is another moving part that can break.
The solution was to create a separate virtual network interface which only the Consul services would use. I could then statically assign an IP to the nodes when they start and those IPs would be accessible by any service on the network.
To test this, I created the interface (see below for a permanent solution):
sudo ifconfig eth0:1 10.0.0.1 netmask 255.255.255.0 up
Then I ran a
hello-world app bound to that IP:
docker run --rm -t -i -p 10.0.0.1:80:80 tutum/hello-world
I then navigated to
http://10.0.0.1/ and saw the hello world page. This is good, it means
eth0:1 works. I wasn't able to create a
docker0:1 interface without issues. As of yet I'm not sure why.
The next step is to launch Consul bound to that IP and then have other services connect to Consul now that the IP is known. For Node.js, I'm using node-consul. Once it knows the Consul server location, it should be able to pull all the relevant services.
To start Consul:
# This is only an example snipped from a larger file
docker run -d -h $HOSTNAME \
--name $NAME \
-p $NETADDR:8400:8400 \
-p $NETADDR:8500:8500 \
-p $NETADDR:8600:53/udp \
progrium/consul -server -bootstrap -ui-dir /ui
Now you can connect to 10.0.0.1 knowing that Consul is there.
A more permanent network interface
For a more permanent network interface, edit
/etc/network/interfaces and drop in the following at the bottom:
# Private network for Consul services
iface eth0:1 inet static
Then bring up the interface:
$ sudo ifup eth0:1
Always check that the interface came up ok, as ifconfig/ifup/ifdown are very strange beings who do what they want, when they want, usually in a way you don't want.
This is less of a structured post and more of a notes-as-I-go format post. Please don't follow these instructions like they will build you a system. They will not. But what they might do is give you an idea as to how you can work certain things. Most of all, they will serve as a great resource for future me.
Today I've been playing with UFW, Ubuntu, OpenVPN, and Docker. A few other things as well but this post relates to those technologies, mainly UFW, Ubuntu, and Docker.
In case you didn't know, like I didn't know, when you expose ports, Docker thinks a great idea would be to put some rules into iptables to allow the traffic to pass through. I'm sure there is a good reason for this, but you may find that when you enable UFW, traffic still gets through. Now that I understand what is going on, it all makes sense, but getting to that point has taken up my evening.
I followed this tutorial on how To Run OpenVPN in a Docker Container on Ubuntu 14.04 which was a breeze. Really highlighted the strengths of Docker. This setup assigns you an IP from 192.168.255.0/24.
My next step, now that I had VPN access, was to shut down eth0 on the machine, allowing through only ports 22 & 1194/udp. What I didn't know was that Docker was going to be a royal pain in the arse during this time. It would continue to bypass my rules and make me wonder what the hell was wrong with UFW. Nothing was, it was Docker playing with iptables.
To stop Docker from doing this, you have to start the Docker daemon, or Docker engine I think it's now called, with the --iptables=false flag. To do this, on Ubuntu 14.04, open up
/etc/default/docker in your favourite text editor and add the following line:
Save that, and then restart the Docker daemon/engine/server thing:
sudo restart docker
Now, I'm not sure what happens here with existing containers, but I went ahead and deleted them and started fresh. Now when you start new containers, there won't be crazy rules bypassing UFW.
For UFW, I had to add a few rules. I added in ports 22 (SSH) and 1194/udp (VPN). I also added a rule to allow all traffic from docker0:
sudo ufw allow 22
sudo ufw allow 1194/udp
sudo ufw allow in on docker0 to any
sudo ufw enable
Next, I started some containers, and tried to access them. No access... turned on the VPN, tried again, and bingo. I had access. It took a while to get there but I got there. Secured access to my containers.
Now I need a long, hot bath to relax. Thanks a lot Docker!
To get the container ID from within a container, use this:
$ cat /proc/self/cgroup | grep "docker" | sed s/\\//\\n/g | tail -1
I'll probably stick it in an npm module at some point.
I was just reading through one of Progrium's scripts when I came across
set -eo pipefail at the beginning of a script. Having not seen that before, I decided to Google. This is the result of that.
You can use
set to manipulate shell variables and functions. Some of these can help you write safer scripts.
If any command fails,
set -e will make the entire script fail, rather than just skipping onto the next line. If you want to allow a line to fail then you can pop
|| true onto the end of it.
This will treat unset variables as an error, and immediately exit the script.
set -o pipefail
By default only the last command in a list of piped commands returns a failure code if it fails. By using
set -o pipefile, if any of the commands fail, the line will fail. Using this with
set -e means that if any command in a piped command fails, the script will fail.
Now back to my reading...
To install Consul on Ubuntu 14.04, first make sure you have unzip available:
$ apt-get install -y unzip
Now, grab the Consul archive, make sure to get the latest & the right architecture, at the time of writing it is 0.5.0, and for Ubuntu it's linux_amd64:
$ wget https://dl.bintray.com/mitchellh/consul/0.5.0_linux_amd64.zip
Now unzip it.
$ unzip 0.5.0_linux_amd.zip
consul to somewhere in your PATH:
$ mv consul /usr/bin/local/consul
Finally, check it works:
$ consul --version
Consul Protocol: 2 (Understands back to: 1)