adamkdean

software engineering

Pipe Stream to Express

By Adam K Dean on

As part of a project I'm working on, I need to grab some data via HTTP/S and transmit it as binary, but with access to it's headers. The following is a quick proof of concept to listen for HTTP requests with Express, request an external image on request, and pipe the response back through to the Express response socket while having access to response metadata such as headers.

'use strict'

const express = require('express')
const request = require('request')
const through2 = require('through2')

const app = express()
const imageUrl = 'http://via.placeholder.com/800x600?text=example'

app.use((incomingRequest, outgoingResponse) => {
  const outgoingRequest = request(imageUrl)
  const bufferOnPipe = through2(function (chunk, enc, callback) {
    this.push(chunk)
    callback()
  })
  const bufferedResponse = outgoingRequest.pipe(bufferOnPipe)

  outgoingRequest.on('response', (incomingResponse) => {
    if (incomingResponse.statusCode === 200) {
      console.log('statusCode:', incomingResponse.statusCode)
      console.log('headers:', incomingResponse.headers)
      bufferedResponse.pipe(outgoingResponse)
    } else {
      console.log('non-200 statusCode:', incomingResponse.statusCode)
    }
  })
})

app.listen(8000, () => {
  console.log('listening on 8000')
})

Hiatus / 5th Gen Blog

By Adam K Dean on

It's been over 12 months since my last blog post, which was back on akd.io before that domain expired. I'd moved most of the blog posts over from adamkdean.co.uk and let that domain expire, and now here we are, on akd.sh with all the old blog posts.

The last year or two have been a really busy time for me, work aside; had a baby, bought a house, spent two months refurbishing that house, time really does seem stretched these days, but I've been able to put this blog together in an hour or two tonight using one of the products I work on @DADI. I used to build each blog myself, but that became a bit boring after the first couple, generally, I lost interest in such mundane tasks.

So, here I am, the 5th generation of my blog, bringing back some popular hits such as Cross thead calls made easy, No definition found for Table yahoo.finance.quotes, and Reclaim your inodes by deleting dangling docker volumes.

New topics will probably focus around always-existed-only-now-people-care emerging technologies such as blockchains, cryptography, and decentralised applications.

base64 decode explained

By Adam K Dean on

This is part 2 of a series on base64 encoding/decoding, based on a kata I recently completed at codewars. I've used base64 a lot but never have I delved into it enough to understand exactly what goes on. So I took the time to explain via inline comments. I hope you enjoy reading it as much as I enjoyed writing it. Part 1 - base64 encode explained.

String.prototype.fromBase64 = function () {
  const base64chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdef' +
                      'ghijklmnopqrstuvwxyz0123456789+/';
  let result = '',
      encoded = '';

  // step 1. convert base64chars into reverse lookup object
  const base64inv = {};
  for (let i = 0; i < base64chars.length; i++)
    base64inv[base64chars[i]] = i;

  // step 2. remove any characters that are not base64 or padding
  const base64regex = new RegExp(`[^${base64chars}=]`, 'g');
  encoded = this.replace(base64regex, '');

  // step 3. replace padding at the end with A (remember, A equals zero)
  const onePadding = encoded.charAt(encoded.length - 1) === '=';
  const twoPadding = encoded.charAt(encoded.length - 2) === '=';  
  const padding = onePadding ? twoPadding ? 'AA' : 'A' : '';
  encoded = encoded.substring(0, encoded.length - padding.length) + padding;

  // step 4. iterate over the encoded string, four characters at a time
  for (let i = 0; i < encoded.length; i += 4) {

    // step 5. convert the four base64 characters into 6-bit numbers using
    //         the base64 character -> 6-bit number map above
    const dn = base64inv[encoded.charAt(i)];
    const en = base64inv[encoded.charAt(i + 1)];
    const fn = base64inv[encoded.charAt(i + 2)];
    const gn = base64inv[encoded.charAt(i + 3)];

    // step 6. convert these four 6-bit numbers into  one 24-bit number
    //
    // if you remember from before, we split a 24-bit number into four 6-bit numbers:
    //
    // e.g.   00001111 00000101 00001010
    // to     |----||- ---||--- -||----|
    //         d     e      f      g
    //
    // d =    00000000 00000000 00000011
    // e =    00000000 00000000 00110000
    // f =    00000000 00000000 00010100
    // g =    00000000 00000000 00001010
    //
    //
    // we need to left shift them (<<) so that they all line up
    // 
    // 00000000 00000000 00XXXXXX (we have four of these)
    // DDDDDDEE EEEEFFFF FFGGGGGG (we want one of these)        
    const d = dn << 18;      // DDDDDD00 00000000 00000000
    const e = en << 12;      // 000000EE EEEE0000 00000000
    const f = fn << 6;       // 00000000 0000FFFF FF000000
    const g = gn;            // 00000000 00000000 00GGGGGG
    const n = d + e + f + g; // DDDDDDEE EEEEFFFF FFGGGGGG (yay!)

    // step 7. split this 24-bit number into three 8-bit (ASCII) characters
    //
    // if you remember, we had three of these (8-bit): 00000000
    // and we actually wanted one of these (24-bit): 00000000 00000000 00000000
    //
    // to get this, we shift the first number 16 bits left, and the second, 8 bits left:
    //
    // 00000000 <------- -------- first char << 16
    //          00000000 <------- second char << 8
    //                   00000000 third char (no shift)
    //
    // so now we want to reverse this and reclaim our three 8-bit (ASCII) characters,
    // we can do this by shifting the numbers back over to the right, and applying a
    // 255 value logical AND (&) bit mask to ignore anything in the 16 bits on the left
    //
    // e.g.   00001111 00000101 00001010
    // to     |------| |------| |------|
    //         a        b        c
    //
    // >>> 16 00000000 00000000 00001111
    // & 255  00000000 00000000 11111111
    // a =    00000000 00000000 00001111
    //                            
    // >>> 8  00000000 00001111 00000101
    // & 255  00000000 00000000 11111111
    // b =    00000000 00000000 00000101
    //
    // noop   00001111 00000101 00001010
    // & 255  00000000 00000000 11111111
    // c =    00000000 00000000 00001010
    const a = (n >>> 16) & 255;
    const b = (n >>> 8) & 255;
    const c = n & 255;

    // step 8. turn these three 8-bit numbers into ASCII characters, and append to result
    result += String.fromCharCode(a, b, c);
  }

  // step 8. finally, remove any padding that was previously added to make this a multiple of 3
  return result.substring(0, result.length - padding.length);
};

base64 encode explained

By Adam K Dean on

This is from a kata I recently completed at codewars. I've used base64 a lot but never have I delved into it enough to understand exactly what goes on. So I took the time to explain via inline comments. I hope you enjoy reading it as much as I enjoyed writing it. Part 2 - base64 decode explained.

String.prototype.toBase64 = function () {  
  const base64chars = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdef' +
                      'ghijklmnopqrstuvwxyz0123456789+/';

  let plaintext = this,
      result = '',
      padding = '';

  // step 1. ensure that the string is a multiple of three chars
  if (plaintext.length % 3 > 0) {
    for(let count = 0; count < 3; count++) {
      padding += '=';
      plaintext += '\0';
    }
  }

  // step 2. iterate over the input string, three chars at a time
  for (let i = 0; i < plaintext.length; i += 3) {
    // step 3. take three 8-bit (ASCII) chars, and store
    //         them as one single 24-bit number
    //
    // 00000000 8-bit (1 byte)
    // 00000000 00000000 00000000 24-bit (3 bytes)
    //
    // we'll bitshift the first number two bytes (16 bits)
    // we'll bitshift the second number one byte (8 bits)
    // and we'll pop the other number into the third byte
    //
    // 00000000 <------- -------- first char << 16
    //          00000000 <------- second char << 8
    //                   00000000 third char (no shift)    
    const a = plaintext.charCodeAt(i) << 16;
    const b = plaintext.charCodeAt(i + 1) << 8;
    const c = plaintext.charCodeAt(i + 2);
    const n = a + b + c;

    // step 4. separate this 24-bit number into four 6-bit numbers
    //
    // we'll do this by shifting to the right (with zero pad >>>)
    // and then doing a logical AND (&) to strip everything after 
    // the first (right most) six bits
    //
    // e.g.   00001111 00000101 00001010
    // to     |----||- ---||--- -||----|
    //         d     e      f      g
    //
    // >>> 18 00000000 00000000 00000011
    // & 63   00000000 00000000 00111111
    // d =    00000000 00000000 00000011
    //                            
    // >>> 12 00000000 00000000 11110000 
    // & 63   00000000 00000000 00111111
    // e =    00000000 00000000 00110000
    //
    // >>> 6  00000000 00111100 00010100 
    // & 63   00000000 00000000 00111111
    // f =    00000000 00000000 00010100
    //
    // noop   00001111 00000101 00001010
    // & 63   00000000 00000000 00111111
    // g =    00000000 00000000 00001010
    const d = (n >>> 18) & 63;
    const e = (n >>> 12) & 63;
    const f = (n >>> 6) & 63;
    const g = n & 63;

    // step 5. use the four 6-bit numbers as indices to pluck
    //         chars from our char array above, and add to the
    //         result string, before continuing with the loop
    //
    // this works because a 6-bit number is 0-63, and we have
    // 64 characters above. Therefore, we can pluck out chars
    result += base64chars[d];
    result += base64chars[e];
    result += base64chars[f];
    result += base64chars[g];
  }

  // step 6. finally, we'll remove the zero pad we added above
  //         and add the actual padding, then return this string
  return result.substring(0, result.length - padding.length) + padding;
};

List all files changed in last commit

By Adam K Dean on

List all files changed in the last commit by using git diff-tree on HEAD.

git diff-tree --no-commit-id --name-only -r HEAD

For example, if you changed README.md in your last commit:

adam@macbook:project (master) $ git diff-tree --no-commit-id --name-only -r HEAD
README.md