adamkdean

software engineering

Determine volume location on Docker host

By Adam K Dean on

Yesterday I was having an issue that I've had before. It stumped me last time and it stumped me again. I just didn't think and it cost me too much time.

The problem I was having, a problem I'd had before, is that I run Jenkins in a container with the Docker host socket passed through, so when Jenkins creates containers, they actually run on the host alongside it. This is great, but when you need to directly map files from the workspace to the container, things don't act the way you may expect.

This is because the host is running the docker commands, Jenkins simply talks to the host via the socket. So those files that are in /jenkins/jobs/yourjob/workspace/... won't be in that location on the host.

The way I run Jenkins is that I have a data volume container called jenkins_data which as you've guessed, holds the data. In order to use a -v src:dest volume map, I'm going to need to locate these files on the host. Docker inspect helps here.

docker inspect --format '{{ range .Mounts }}{{ if eq .Destination "/jenkins" }}{{ .Source }}{{ end }}{{ end }}' jenkins

Once I have the location, I need to do a search and replace on the $WORKSPACE variables which Jenkins exposes. After that, I have the location of the /jenkins volume on the host.

JENKINS_ROOT=$(docker inspect --format '{{ range .Mounts }}{{ if eq .Destination "/jenkins" }}{{ .Source }}{{ end }}{{ end }}' jenkins)
JENKINS_PATH=$(echo $WORKSPACE | sed "s|^\/jenkins|$JENKINS_ROOT|g")

Now I have $JENKINS_PATH I can directly map files or directories in from within my container.

docker run d \
    -v $JENKINS_PATH/some/file.js:/var/lib/app/file.js \
    -v $JENKINS_PATH/dir:/var/lib/app/dir \
    someapp

Check if temp table exists in SQL

By Adam K Dean on

I'm going to start posting some SQL snippets, either things I learn day to day or things that I've had sat around for years.

Today's is a checking if a temp table exists (and then dropping it.)

IF OBJECT_ID('Tempdb..#tablename') IS NOT NULL
BEGIN
    -- do something here, like say, drop that table?
    DROP TABLE #tablename;
END

Reclaim your inodes by deleting dangling docker volumes

By Adam K Dean on

Earlier today I had a strange issue where Docker builds started failing, citing there was no free space. When I looked, there was gigabytes of free space. I looked into this further and found that Jenkins liked to open a lot of files. 30,000 for a fresh install, in fact. I thought this was my problem but it wasn't. Something else was causing the system to run out of space.

I dug around a bit more and found that the problem was inodes, which are data structures used to represent filesystem objects such as files or directories. To be more precise, the problem was a distinct lack of available inodes. Of the 2.9 million available, 2.9 million were in use. That's 100%.

I had a funny feeling that dangling volumes had something to do with this, so I had a look at how many I had:

$ docker images -qf 'dangling=true' | wc -l
100

These were quite large images too, comprising of multiple files and directories. So many in fact that it was using up all the inodes available on the system. To repeat, that was 2.9M inodes.

I was able to clear it up by running docker images -qf 'dangling=true' | xargs docker rmi, which takes a while but cleared it up. In case this happens in the future, I've put some logging in place to keep an eye on the inode usage, and created a little script for people to use in case I'm not around and this happens again.

Here is can-i-have-my-inodes-back-please.sh:

#!/bin/bash
#
# Are your inodes all tied up?
# Would you like to make them liquid again?
# Run this script for up to instant relief
# (smallprint: maytakeuptotwentyminutesdependingonnumberofdanglingvolumes)

DANGLING_NUM=$(docker images -qf 'dangling=true' | wc -l)

read -p "Are you sure you want to remove $DANGLING_NUM dangling volumes? (Y/N) " prompt

if [[ $prompt == "y" || $prompt == "Y" || $prompt == "yes" || $prompt == "Yes" ]]; then
  docker images -qf 'dangling=true' | xargs docker rmi
else
  exit 0
fi

And in case you're interested, the crontab runs this script hourly, log-inode.sh:

#!/bin/bash
INODE=$(df -i | sed -n 2p | awk '{ print $5 }')
DATE=$(date +'%m/%d/%Y %H:%M')
echo "$DATE,$INODE"

In a few weeks I'll grab that csv data and bung it into Excel and see just how bad this inode problem is. Maybe a nightly dangling volume cleanup will do the trick.

Add notices to auto-generated files with gulp

By Adam K Dean on

I wrote a little tool today to pop into my gulp workflow which writes some text to the top of files processed by gulp. The problem I find is that quite often you can be working on a project with source files and generated files (e.g. jsx -> js) and accidentally edit the wrong file, only for your changes to be overwritten.

This tool will put a notice at the top of auto-generated files to hopefully help put a stop to that. There are other packages that do this too, but so far in the past week there have been about 10 releases of gulp-header and quite a few of them were broken. Also they originated from godaddy so confidence in the package is pretty low!

Usage

You can use the default notice:

var gulp = require('gulp'),
    notice = require('gulp-notice');

gulp.task('default', function () {
    gulp.src('src/*.js')
        .pipe(notice())
        .pipe(gulp.dest('dist/'));
});

Which will prepend files with:

/* --------------------------------------------------------------------- *\
|  This code was auto-generated by a tool.                                |
|                                                                         |
|  Changes to this file may cause incorrect behavior and will be lost if  |
|  the code is regenerated.                                               |
\* --------------------------------------------------------------------- */

Or you can provide your own string as the first parameter:

var gulp = require('gulp'),
    notice = require('gulp-notice');

var text = '/* this file was auto-generated */';

gulp.task('default', function () {
    gulp.src('src/*.js')
        .pipe(notice(text))
        .pipe(gulp.dest('dist/'));
});

Which will prepend files with:

/* this file was auto-generated */

Working wth streams

If you're working with streams (e.g. vinyl-source-stream) then gulp-streamify will help you. Let's say you're using browserify with vinyl-source-stream, you can wrap gulp-notice with streamify and it'll work:

return b.bundle()
    .pipe(source(entry))
    .pipe(streamify(notice()))
    .pipe(gulp.dest('./public/js'));

Install it

npm install gulp-notice

GitHub: https://github.com/adamkdean/gulp-notice
NPM: https://www.npmjs.com/package/gulp-notice

Koa middleware for serving static files

By Adam K Dean on

Quite often I find myself using the same snippet of code over and over in projects for serving up static files as part of projects that use Koa. Today I exceeded my limit for copy pasting, so I've bundled this together into a module called koa-serve.

It's probably a lot simpler than koa-static but it works well for what I want and need.

var koa = require('koa'),
    serve = require('koa-serve'),
    app = koa();

app.use(serve('assets'));
app.listen(8000);

You can also define where you root dir is, if isn't __dirname.

var koa = require('koa'),
    serve = require('koa-serve'),
    app = koa();

app.use(serve('assets', '/path/to/your/root'));
app.listen(8000);

Example if your client files are in the parent directory, and index.js is in server/ for example:

var koa = require('koa'),
    serve = require('koa-serve'),
    path = require('path'),
    app = koa();

app.use(serve('assets', path.join(__dirname, '..', 'client'));
app.listen(8000);

Install it

npm install koa-serve

GitHub: https://github.com/adamkdean/koa-serve
NPM: https://www.npmjs.com/package/koa-serve