Reclaim your inodes by deleting dangling docker volumes

Published Tuesday, 8 September 2015

Earlier today I had a strange issue where Docker builds started failing, citing there was no free space. When I looked, there was gigabytes of free space. I looked into this further and found that Jenkins liked to open a lot of files. 30,000 for a fresh install, in fact. I thought this was my problem but it wasn't. Something else was causing the system to run out of space.

I dug around a bit more and found that the problem was inodes, which are data structures used to represent filesystem objects such as files or directories. To be more precise, the problem was a distinct lack of available inodes. Of the 2.9 million available, 2.9 million were in use. That's 100%.

I had a funny feeling that dangling volumes had something to do with this, so I had a look at how many I had:

$ docker images -qf 'dangling=true' | wc -l

These were quite large images too, comprising of multiple files and directories. So many in fact that it was using up all the inodes available on the system. To repeat, that was 2.9M inodes.

I was able to clear it up by running docker images -qf 'dangling=true' | xargs docker rmi, which takes a while but cleared it up. In case this happens in the future, I've put some logging in place to keep an eye on the inode usage, and created a little script for people to use in case I'm not around and this happens again.

Here is

# Are your inodes all tied up?
# Would you like to make them liquid again?
# Run this script for up to instant relief
# (smallprint: maytakeuptotwentyminutesdependingonnumberofdanglingvolumes)

DANGLING_NUM=$(docker images -qf 'dangling=true' | wc -l)

read -p "Are you sure you want to remove $DANGLING_NUM dangling volumes? (Y/N) " prompt

if [[ $prompt == "y" || $prompt == "Y" || $prompt == "yes" || $prompt == "Yes" ]]; then
  docker images -qf 'dangling=true' | xargs docker rmi
  exit 0

And in case you're interested, the crontab runs this script hourly,

INODE=$(df -i | sed -n 2p | awk '{ print $5 }')
DATE=$(date +'%m/%d/%Y %H:%M')
echo "$DATE,$INODE"

In a few weeks I'll grab that csv data and bung it into Excel and see just how bad this inode problem is. Maybe a nightly dangling volume cleanup will do the trick.

Add notices to auto-generated files with gulp

Published Tuesday, 25 August 2015

I wrote a little tool today to pop into my gulp workflow which writes some text to the top of files processed by gulp. The problem I find is that quite often you can be working on a project with source files and generated files (e.g. jsx -> js) and accidentally edit the wrong file, only for your changes to be overwritten.

This tool will put a notice at the top of auto-generated files to hopefully help put a stop to that. There are other packages that do this too, but so far in the past week there have been about 10 releases of gulp-header and quite a few of them were broken. Also they originated from godaddy so confidence in the package is pretty low!


You can use the default notice:

var gulp = require('gulp'),
    notice = require('gulp-notice');

gulp.task('default', function () {

Which will prepend files with:

/* --------------------------------------------------------------------- *\
|  This code was auto-generated by a tool.                                |
|                                                                         |
|  Changes to this file may cause incorrect behavior and will be lost if  |
|  the code is regenerated.                                               |
\* --------------------------------------------------------------------- */

Or you can provide your own string as the first parameter:

var gulp = require('gulp'),
    notice = require('gulp-notice');

var text = '/* this file was auto-generated */';

gulp.task('default', function () {

Which will prepend files with:

/* this file was auto-generated */

Working wth streams

If you're working with streams (e.g. vinyl-source-stream) then gulp-streamify will help you. Let's say you're using browserify with vinyl-source-stream, you can wrap gulp-notice with streamify and it'll work:

return b.bundle()

Install it

npm install gulp-notice

More links


Koa middleware for serving static files

Published Wednesday, 12 August 2015

Quite often I find myself using the same snippet of code over and over in projects for serving up static files as part of projects that use Koa. Today I exceeded my limit for copy pasting, so I've bundled this together into a module called koa-serve.

It's probably a lot simpler than koa-static but it works well for what I want and need.

var koa = require('koa'),
    serve = require('koa-serve'),
    app = koa();


You can also define where you root dir is, if isn't __dirname.

var koa = require('koa'),
    serve = require('koa-serve'),
    app = koa();

app.use(serve('assets', '/path/to/your/root'));

Example if your client files are in the parent directory, and index.js is in server/ for example:

var koa = require('koa'),
    serve = require('koa-serve'),
    path = require('path'),
    app = koa();

app.use(serve('assets', path.join(__dirname, '..', 'client'));

Install it

npm install koa-serve

More links


Node.js IsDirectory

Published Wednesday, 12 August 2015

Here's a useful snippet for checking if a path is a directory in Node.js, this obviously requires('fs').


Also available to you:


Useful git aliases

Published Friday, 31 July 2015

I've been so busy recently that this blog has been neglected. I'm planning on fixing some design quirks and adding in some archiving, maybe even giving it a biy of a face lift so it's not so drab. Something I don't want to happen is to have empty months when it comes to my archive so today is the last day of July and I'll show you what aliases I have in my git config.

    a = "!git add -A . && git add -u"
    ab = branch -avv
    dt = difftool
    s = status
    b = branch -a
    k = !gitk --all --select-commit=HEAD &
    r = remote -v
    rh = reset --hard
    pr = "!git add . && git add -u && git stash && git pull && git stash pop"
    logp = log -n10 --decorate --graph --pretty=tformat:\"%C(yellow)%h%Creset%C(cyan)%d%C(reset) %s %C(green)(%an, %ar)%C(reset)\" --relative-date
    diffp = diff --ignore-space-at-eol -b -w   
    llm = log -1
    unadd = reset --
    ua = reset --
    h = rev-list --max-count=1 HEAD
    list-merge-backups = !git status -su | grep -e'\\.orig$' | cut -f2 -d' '
    delete-merge-backups = !git list-merge-backups | xargs rm
    mergefrom = merge -s recursive -X theirs

Well, that was an easy blog post, but I can't use git without these, they save a lot of time!

Showing posts 1-5 of 170