Create a jq Docker Image with Automated Build

I have created a jq Docker image based on BusyBox with automated builds. BusyBox is really really small in size, so the jq image I have created is also very small, just a little over 6 MB.

Here is the source code and image repositories:

Building Docker image is pretty straight-forward, just follow the instruction:

https://docs.docker.com/userguide/dockerrepos/#automated-builds

I have also created a tag v1.4 in my GitHub repository to match the release of jq binary. This should also be reflected in Docker registry. After couple tries, here is the build details for adding both latest and 1.4 tags:

1
2
3
4
Type Name Dockerfile Location Tag Name
------------------------------------------------
Tag v1.4 / 1.4
Branch master / latest

The first two columns match git branch and tag, and the last column reflects Docker tags. See https://github.com/realguess/docker-jq/tags and https://registry.hub.docker.com/u/realguess/jq/tags/manage/.

This is done by starting an automated build with type of tag instead of branch. Everything is done via the Docker Hub website.

If I pull down this repository:

1
$ docker pull realguess/jq

It should give me two image layers with the two different image IDs, which makes sense, as latest commit usually is not the same as the tagged one.

After a while, the index should be built, and I can search it via:

1
2
3
$ docker search jq
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
realguess/jq 1 [OK]

The image is listed as AUTOMATED, but the description is missing here in the search command.

There are two types of descriptions:

  • Short description
  • Full description

Full description will be generated automatically via README.md file. I thought the short description can also be generated via README-short.txt, however, this is not the case. You can add it in the settings page for example:

https://registry.hub.docker.com/u/realguess/jq/settings/

Automated builds are triggered automatically with GitHub and BitBucket repository. Once any commit is pushed to either repository, a trigger will be sent to Docker Hub, and automated build will start.

Synchronous Console Methods

In Node v0.10.x, the console functions are synchronous when the destination is a terminal or a file (to avoid lost messages in case of premature exit) and asynchronous when it’s a pipe (to avoid blocking for long periods of time). See console. We can test it with the following script:

Stumbled Upon Lean Coffee

Stumbled upon Lean Coffee. Lean movement adventures into coffee brewing? No, it is about structured, but agenda-less meeting with participants generated topics, or lean meeting. It is not truly agenda-less, but the topics of the meeting agenda are participant-sourced and participant-judged or voted. So, instead of top-down, but more democratic and bottom-up approach.

Use CNAME for Resovling Private and Public IP Address in Amazon EC2

“A security group acts as a virtual firewall that controlls the traffic for one or more instances.” 1 The Amazon EC2 Security Groups are not just capable controlling traffic from an IP address, but also from all EC2 instances belong to a specific security group. I want to allow an instance belonging to one security group to access an instance belongs to another security group via a custom domain name (subdomain.example.com).

But when I configured the subdomain via Amazon Route 53, I have misconfigured it by assigning a A record, an IP address or the Elastic IP address of the instance. I should have used CNAME, and assigned the public DNS (the public hostname of the EC2 instance).

Patch ShellShock Vulnerability

ShellShock is a Bash vulnerability, which allows bypassing environment restrictions in certain environments. See Ubuntu Security Notice USN-2363-1. And for more detailed description, see CVE-2014-7169.

This Bash bug can be potentially bigger than the Heartbleed bug, because Bash shell is usually the default shell for the user, and many commands will also spawn Bash to execute. See this answer http://askubuntu.com/a/528102 from StackOverflow.

Running the following command will determine if the system is vulnerable or not:

1
$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

Source: http://t.co/nfDCUdRnb5

Luckily, this is relatively easy to patch. Simple apt-get update && apt-get upgrade will patch the Bash with the latest security fix. For example, this is a patched Bash version:

1
2
$ bash --version
GNU bash, version 4.2.25(1)-release

This is for Ubuntu 12.04 LTS. If you are running Ubuntu 14.04 LTS, you have to
upgrade to Bash 4.3-7ubuntu1.3.

Dirname Does Not Follow Symlink

Node’s global object __dirname returns the name of the directory that the currently executing script resides in. However, if a directory is symbolic linked or a symlink, it still return the original path.

For example, the following directory structure:

1
2
3
4
5
6
7
8
/tmp/dirname/
├── foo
│ └── app.js
└── lib
├── bar -> ../foo
└── util.js
3 directories, 2 files

The app.js contains the following line:

1
console.log(__dirname);

Run the script:

1
2
$ cd /tmp/dirname/foo && node app.js
/tmp/dirname/foo

You get the directory the app.js script is residing it. But if you do the same from the symlinked directory:

1
2
$ cd /tmp/dirname/lib/bar && node app.js
/tmp/dirname/foo

Well, you get the same answer, even the current working directory is different:

1
2
$ cd /tmp/dirname/lib/bar && pwd
/tmp/dirname/lib/bar

The return of __dirname does not follow symbolic link.

So, have to be extra careful when requiring files that is symbolic linked. If you think you are in lib/bar directory and try to require the util.js script in app.js:

1
2
3
var path = require('path');
console.log(require('../util'));
console.log(require(path.join(__dirname, '..', 'util')));

Both statements will throw module not found error. Path join does not help either.

The best practice is to avoid symlinks and relative directory requiring. Instead, set up required as modules, and install them in node_modules directory, then they will become top-level modules.

Non-Interactive Redis Install

To build Redis from source, we first need to install TCL 8.5 or newer, this is needed for make test later:

$ sudo apt-get install -y tcl

Now clone and make:

$ git clone https://github.com/antirez/redis
$ git checkout 2.8.13
$ make
$ make test
$ sudo make install
$ sudo utils/install_server.sh

Binary (redis-cli and redis-server) will be installed into /usr/local/bin.

The last command with utils/install_server.sh is an interactive command. Since the script is a shell script using the read built-in command and -p option for prompt, we can make it non-interactive by redirecting input from echo command:

$ echo -n | sudo utils/install_server.sh

Without pumping any value into the script, the default values are used.

If really want to customize it, we can add our own values:

$ echo -e \
  "${PORT}\n${CONFIG_FILE}\n${LOG_FILE}\n${DATA_DIR}\n${EXECUTABLE}\n" | \
  sudo utils/install_server.sh

There are 6 read statements, hence n - 1 newline characters. Without using -n, the last newline character is supplied by echo.

Here are the default values:

PORT=6379
CONFIG_FILE=/etc/redis/6379.conf
LOG_FILE=/var/log/redis_6379.log
DATA_DIR=/var/lib/redis/6379
EXECUTABLE=/usr/local/bin/redis-server

The utils/install_server.sh script should return something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Welcome to the redis service installer
This script will help you easily set up a running redis server
Selecting default: 6379
Selected default - /etc/redis/6379.conf
Selected default - /var/log/redis_6379.log
Selected default - /var/lib/redis/6379
Selected config:
Port : 6379
Config file : /etc/redis/6379.conf
Log file : /var/log/redis_6379.log
Data dir : /var/lib/redis/6379
Executable : /usr/local/bin/redis-server
Cli Executable : /usr/local/bin/redis-cli
Copied /tmp/6379.conf => /etc/init.d/redis_6379
Installing service...
System start/stop links for /etc/init.d/redis_6379 already exist.
Success!
/var/run/redis_6379.pid exists, process is already running or crashed
Installation successful!

Set an alias for Redis client:

$ cd /usr/local/bin && sudo ln -s redis-cli redis

For more advanced install, see the README file.

Gulp Task Alias

Gulp API does not have a method to create an alias task. But, there are two ways to do this now:

  1. Create a task alias by dependency
  2. Create a task alias by duplication

Dependency

Gulp dependency system can be used to create an alias of a task:

gulp.task('task', function(){});
gulp.task('alias', ['task']);

Here, task alias is the alias of the task task. When list the tasks, it shows the dependency tree:

$ gulp --tasks
Tasks for ~/gulpfile.js
├── task
└─┬ alias
  └── task

When running the tasks, the dependency is obvious:

$ gulp alias
Starting 'task'...
Finished 'task' after 49 μs
Starting 'alias'...
Finished 'alias' after 6.83 μs

Duplication

Another way to create a task alias is by duplicating the task definition:

var task = function () {};    
gulp.task('task' , task);
gulp.task('alias', task);

But in this way, there is no implicit relationship that will indicates that one task is an alias of another. As you can see from the task list:

$ gulp --tasks
Tasks for ~/gulpfile.js
├── task
└── alias

To Gulp, there are two different tasks. But when the alias task is run, the console log is cleaner than the dependency one:

$ gulp alias
Starting 'alias'...
Finished 'alias' after 45 μs

Gulp is still involving, therefore, API might change regarding for creating alias of a task. At the moment, I like the second option. Alias and dependency are two different concepts. Dependency should be left as dependency. Alias is just another name to run the same task.

Extended Pattern Matching

Bash supports extended pattern matching. By using the built-in utility we can check if it is enabled or not:

$ shopt extglob

If not, to enable it:

$ shopt -s extglob

By default extglob is on in interactive shells, but off in non-interactive shells.

The key about extended pattern matching is pattern list via | (what we usually see as a OR operator). But do not think about it as that, think about it as a list of array pattern that are separated by | instead of ,. And actually one of them is the same as {} or brace expansion, but it can do more than expanding.

Here are the pattern operators:

? * + @ !

Create some example files:

$ touch a{,1,2,11,12}.js && ls
a11.js  a12.js  a1.js  a2.js  a.js

?(pattern-list)

Zero or one (any one) occurrence of the giving pattern:

$ ls a?(2|1).js
a1.js  a2.js  a.js

*(pattern-list)

Zero or more occurrences of the giving pattern (essentially everything):

$ ls a*(2|1).js
a11.js  a12.js  a1.js  a2.js  a.js

+(pattern-list)

One or more of the giving pattern (notice a.js is missing):

$ ls a+(2|1).js
a11.js  a12.js  a1.js  a2.js

@(pattern-list)

Any one of giving pattern:

$ ls a@(2|1).js
a1.js  a2.js

!(pattern-list)

None of the giving pattern:

$ ls a!(2|1).js
a11.js  a12.js  a.js

One mistake I had was getting confused between extended pattern matching and brace expansion, for example:

$ ls test/@{src|spec}/*.js
ls: cannot access test/@src/*.js: No such file or directory
ls: cannot access test/@spec/*.js: No such file or directory

Extended pattern uses parentheses () not braces {} as in brace expansion.

Also, these two patterns are the same:

test/@(src|spec)/*.js
test/{src,spec}/*.js

In some situations, extended pattern matching does not work, for example, matching files from the current directory and from one of the subdirectories with the following directory structure:

1
2
3
4
5
6
7
8
9
$ tree
.
├── app.js
├── lib
│ └── util.js
└── test
└── main.js
2 directories, 3 files

I would like to match js files from the current directory and lib/ directory, sort of like:

$ ls *.js lib/*.js
app.js  lib/util.js

But this does not do it:

$ ls @(.|lib)/*.js

Instead, use brace expansion:

$ ls {.,lib}/*.js

Finally, Node’s Minimatch supports brace expansion, extended globbing and globstar.