When I first started writing Dockerfile for building a NodeJS app, I will do:

1
2
3

WORKDIR /app
COPY . /app
RUN npm install

Sure, this is very simple, but every time a change in the source file, the entire dependency tree needs to be re-installed. The only time you need to rebuild is when package.json changes.

One trick is to move package.json elsewhere, and build the dependencies, then move back:

1
2
3
4
5

WORKDIR /app
ADD package.json /tmp
RUN cd /tmp && npm install
COPY . /app
RUN mv /tmp/node_modules /app/node_modules

However, if there are a lot of depending packages with lots of files, the last step will take a long time.

How about using symbolic link to shorten the time?

1
2
3
4
5

WORKDIR /app
ADD package.json /tmp
RUN cd /tmp && npm install
COPY . /app
RUN ln -s /tmp/node_modules /app/node_modules

Well, it works, but not every NPM package is happy about it. Some will complain about the softlink, result in errors.

So, how can we reuse caches as much as possible, and:

1. Do not need to reinstall dependencies
2. Do not use symbolic link

Here is a solution, just move down the COPY statement:

1
2
3
4
5
6
7
8
9
10
11
12
13

# Set the working directory, which creates the directory.
WORKDIR /app
# Install dependencies.
ADD package.json /tmp
RUN cd /tmp && npm install
# Move the dependency directory back to the app.
RUN mv /tmp/node_modules /app
# Copy the content into the app directory. The previously added "node_modules"
# directory will not be overridden.
COPY . /app

With this configuration, if package.json stays the same, only the last cache layer will be rebuilt when source codes change.

This is a Node.js starting script to stream HTTP request directly into response:

1
2
3

require('http').createServer((req, res) => {
  req.pipe(res); // Pipe request directly to response
}).listen(3000);

It behaves almost like an echo, you get back whatever you sent. For example, use HTTPie to make a request to the above server:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

$ echo foo | http --verbose --stream :3000 Content-Type:text/plain
POST / HTTP/1.1
Accept: application/json, */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 4
Host: localhost:3000
User-Agent: HTTPie/0.9.6
Content-Type: text/plain
foo
HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked
foo

We can also add the Content-Type response header to echo back what the entire media type is after assembling all chunks.

1
2
3
4
5
6
7

require('http').createServer((req, res) => {
  req.pipe(res); // Pipe request directly to response
  if (req.headers['content-type']) {
    res.setHeader('Content-Type', req.headers['content-type']);
  }
}).listen(3000);

The response should have the Content-Type field as below:

1
2
3
4
5
6

HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: text/plain
Transfer-Encoding: chunked
foo

Notice that instead of usual Content-Length in the response header, we’ve got Transfer-Encoding: chunked. The default transfer encoding for Node.js HTTP is chunked:

Sending a ‘Content-length’ header will disable the default chunked encoding.[^1]

About transfer encoding:

Chunked transfer encoding is a data transfer mechanism in version 1.1 of the Hypertext Transfer Protocol (HTTP) in which data is sent in a series of “chunks”. It uses the Transfer-Encoding HTTP header in place of the Content-Length header, which the earlier version of the protocol would otherwise require. Because the Content-Length header is not used, the sender does not need to know the length of the content before it starts transmitting a response to the receiver. Senders can begin transmitting dynamically-generated content before knowing the total size of that content. … The size of each chunk is sent right before the chunk itself so that the receiver can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero.[^2]

With the above starting script, now you can attach some transform streams to manipulate the request and stream back in chunked response.

Settings:

1
2
3
4

$ node --version
v6.3.1
$ http --version
0.9.6

[^1]: HTTP, Node.js API Docs

[^2]: Chunked transfer encoding, Wikipedia

One liner to create a Node.js HTTP server:

1

require('http').createServer((req, res) => res.end('OK')).listen(3000);

Also see the previous one liners for older Node.js versions.

Follow up with the previous blog on various methods to install Node.js, here is the updated one with Node.js v4.x and v6.x versions.

Install Node.js v4.x via NodeSource[^1] setup script:

1
2

$ curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
$ sudo apt-get install -y nodejs

sudo -E option indicates to the security policy that the user wishes to preserve their existing environment variables[^2].

With Docker and Dockerfile, sudo should be removed, because root:

1
2

RUN curl -sL https://deb.nodesource.com/setup_4.x | bash - && \
    apt-get install -y nodejs

Install Node.js v6.x via the same method:

1
2

$ curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
$ sudo apt-get install -y nodejs

The above installation method has been tested on:

• Debian 8.5
• Node.js v4.4.7 and v6.2.2

In summary, the methods to install Node.js covered are:

• Package manager
• Nodesource script
• From the source

[^1]: NodeSource provides binary distribution setup and support scripts.
[^2]: See man sudo.

Versions frequently used in NPM package dependencies are caret ranges or ^version. And npm install uses caret ranges as well.

Caret range:

Allows changes that do not modify the left-most non-zero digit in the [major, minor, patch] tuple. In other words, this allows patch and minor updates for versions 1.0.0 and above, patch updates for version 0.x >=0.1.0, and no updates for version 0.0.x.

Most of time, using caret ranges as dependency versions work perfectly fine. But once a while, there are bugs. In one of projects, JS-YAML with caret version ^3.4.3 was used and installed:

1
2
3
4
5

{
  "dependencies": {
    "js-yaml": "^3.4.3"
  }
}

Time passes. When npm install was run again from a freshly cloned project code base, version 3.5.2 was installed. Since JS-YAML version 3.5.0, errors are thrown when safe loading a spec with duplicate keys. If there are no duplicate keys in the YAML file, this will be fine. However, one of the files has it. The correct approach is to actually fix the duplicate keys. But it requires additional work back and forth. You’ve heard this before: “Hey it works when I installed it.”

The point here is to use exact versions instead of letting the package manager decides by dropping the caret character:

1
2
3
4
5

{
  "dependencies": {
    "js-yaml": "3.4.3"
  }
}

This will avoid the above mentioned problem. We can manually update the outdated NPM packages.

Don’t let machine decide. You Do!

Based on the following Node.js statement:

1

require('crypto').createHash(algorithm).update('secret', 'utf8').digest('base64');

On a Ubuntu 14.04 system with OpenSSL:

1
2

$ openssl version
OpenSSL 1.0.1f 6 Jan 2014

List available digest algorithms:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

$ openssl list-message-digest-algorithms
DSA
DSA-SHA
DSA-SHA1 => DSA
DSA-SHA1-old => DSA-SHA1
DSS1 => DSA-SHA1
MD4
MD5
RIPEMD160
RSA-MD4 => MD4
RSA-MD5 => MD5
RSA-RIPEMD160 => RIPEMD160
RSA-SHA => SHA
RSA-SHA1 => SHA1
RSA-SHA1-2 => RSA-SHA1
RSA-SHA224 => SHA224
RSA-SHA256 => SHA256
RSA-SHA384 => SHA384
RSA-SHA512 => SHA512
SHA
SHA1
SHA224
SHA256
SHA384
SHA512
DSA
DSA-SHA
dsaWithSHA1 => DSA
dss1 => DSA-SHA1
ecdsa-with-SHA1
MD4
md4WithRSAEncryption => MD4
MD5
md5WithRSAEncryption => MD5
ripemd => RIPEMD160
RIPEMD160
ripemd160WithRSA => RIPEMD160
rmd160 => RIPEMD160
SHA
SHA1
sha1WithRSAEncryption => SHA1
SHA224
sha224WithRSAEncryption => SHA224
SHA256
sha256WithRSAEncryption => SHA256
SHA384
sha384WithRSAEncryption => SHA384
SHA512
sha512WithRSAEncryption => SHA512
shaWithRSAEncryption => SHA
ssl2-md5 => MD5
ssl3-md5 => MD5
ssl3-sha1 => SHA1
whirlpool

Calculate hashes for each and every uniquely listed algorithm:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

$ openssl list-message-digest-algorithms | \
  cut -d ' ' -f 1 | \
  xargs -n 1 -I {} node -e "algorithm = '{}'; \
  digest = require('crypto').createHash('{}').update('secret', 'utf8').digest('base64'); \
  console.log(algorithm, digest)" | \
  sort -u
DSA 5en6G6MezRroT3XKqkdPOmY/BfQ=
DSA-SHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
DSA-SHA1-old 5en6G6MezRroT3XKqkdPOmY/BfQ=
DSA-SHA 5en6G6MezRroT3XKqkdPOmY/BfQ=
dsaWithSHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
dss1 5en6G6MezRroT3XKqkdPOmY/BfQ=
DSS1 5en6G6MezRroT3XKqkdPOmY/BfQ=
ecdsa-with-SHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
md4WithRSAEncryption Z9Pa/vY/8AYDru83ac+/DQ==
MD4 Z9Pa/vY/8AYDru83ac+/DQ==
md5WithRSAEncryption Xr4ilOzQ4PCOq3aQ0qbuaQ==
MD5 Xr4ilOzQ4PCOq3aQ0qbuaQ==
ripemd160WithRSA zZi/AgLvB+OOh/a9lEXl5zMeLHg=
RIPEMD160 zZi/AgLvB+OOh/a9lEXl5zMeLHg=
ripemd zZi/AgLvB+OOh/a9lEXl5zMeLHg=
rmd160 zZi/AgLvB+OOh/a9lEXl5zMeLHg=
RSA-MD4 Z9Pa/vY/8AYDru83ac+/DQ==
RSA-MD5 Xr4ilOzQ4PCOq3aQ0qbuaQ==
RSA-RIPEMD160 zZi/AgLvB+OOh/a9lEXl5zMeLHg=
RSA-SHA1-2 5en6G6MezRroT3XKqkdPOmY/BfQ=
RSA-SHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
RSA-SHA224 lcf7ypKsUIOv2mKlZKPQFPw7cskUDjy5nqa/Eg==
RSA-SHA256 K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
RSA-SHA384 WKd1ukESvjAFrkQHznV9iP2nHUBJe7gCbsrFTU4//HIyzo3jq1rLMK45dg/ufFPt
RSA-SHA512 vSsar3708Jvp9Szi2NWZZ02Bqp1qRCFpbcTZPdBhnWgs5WtNZKnvCXdhztmeD2cmW192CF5bDufKRpayrW/isg==
RSA-SHA f72qWouzjoc6vqk8vf5NBIVHFXs=
SHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
sha1WithRSAEncryption 5en6G6MezRroT3XKqkdPOmY/BfQ=
SHA224 lcf7ypKsUIOv2mKlZKPQFPw7cskUDjy5nqa/Eg==
sha224WithRSAEncryption lcf7ypKsUIOv2mKlZKPQFPw7cskUDjy5nqa/Eg==
SHA256 K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
sha256WithRSAEncryption K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
sha384WithRSAEncryption WKd1ukESvjAFrkQHznV9iP2nHUBJe7gCbsrFTU4//HIyzo3jq1rLMK45dg/ufFPt
SHA384 WKd1ukESvjAFrkQHznV9iP2nHUBJe7gCbsrFTU4//HIyzo3jq1rLMK45dg/ufFPt
SHA512 vSsar3708Jvp9Szi2NWZZ02Bqp1qRCFpbcTZPdBhnWgs5WtNZKnvCXdhztmeD2cmW192CF5bDufKRpayrW/isg==
sha512WithRSAEncryption vSsar3708Jvp9Szi2NWZZ02Bqp1qRCFpbcTZPdBhnWgs5WtNZKnvCXdhztmeD2cmW192CF5bDufKRpayrW/isg==
SHA f72qWouzjoc6vqk8vf5NBIVHFXs=
shaWithRSAEncryption f72qWouzjoc6vqk8vf5NBIVHFXs=
ssl2-md5 Xr4ilOzQ4PCOq3aQ0qbuaQ==
ssl3-md5 Xr4ilOzQ4PCOq3aQ0qbuaQ==
ssl3-sha1 5en6G6MezRroT3XKqkdPOmY/BfQ=
whirlpool 4GG4emdK44gOFZq1XtNdbF6Kau+sarCDBKUFiAGNN3soY5uxX97t8AbVfkX3tCmObf786vfJLIJqcI/m0RVusw==

Starter for the most common usage.

Create a MD5 hex hash:

1

require('crypto').createHash('md5').update(data, 'utf8').digest('hex');

Create a SHA256 base64 hash:

1

require('crypto').createHash('sha256').update(data, 'utf8').digest('base64');

Calculating the digest varies with encoding:

1
2
3
4
5
6
7
8
9
10
11
12

> data = 'secret'
'secret'
> require('crypto').createHash('sha256').update(data, 'utf8').digest('hex');
'2bb80d537b1da3e38bd30361aa855686bde0eacd7162fef6a25fe97bf527a25b'
> require('crypto').createHash('sha256').update(data, 'utf8').digest('base64');
'K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols='
> require('crypto').createHash('sha256').update(data, 'utf8').digest('binary');
'+¸\rS{\u001d£ãÓ\u0003aªV½àêÍqbþö¢_é{õ\'¢['
> require('crypto').createHash('sha256').update(data, 'utf8').digest();
<Buffer 2b b8 0d 53 7b 1d a3 e3 8b d3 03 61 aa 85 56 86 bd e0 ea cd 71 62 fe f6 a2 5f e9 7b f5 27 a2 5b>
> require('crypto').createHash('sha256').update(data, 'utf8').digest().toString();
'+�\rS{\u001d����\u0003a��V�����qb���_�{�\'�['

Both hex and base64 are commonly used ones.

Needs an input_encoding when update data, otherwise there will be different results:

1
2
3
4
5
6
7
8
9
10

> data = '秘密'
'秘密'
> require('crypto').createHash('sha256').update(data, 'ascii').digest('hex');
'7588bc51fd0ff10db9c66549e5cb7969b9f6b7cf3ccee080137a8eb1aa06e718'
> require('crypto').createHash('sha256').update(data, 'utf8').digest('hex');
'062a2931da683a9897c5a0b597113b1e9fd0d5bfb63e2a5d7c88b724f7f55c02'
> require('crypto').createHash('sha256').update(data, 'binary').digest('hex');
'7588bc51fd0ff10db9c66549e5cb7969b9f6b7cf3ccee080137a8eb1aa06e718'
> require('crypto').createHash('sha256').update(data).digest('hex');
'7588bc51fd0ff10db9c66549e5cb7969b9f6b7cf3ccee080137a8eb1aa06e718'

Don’t assume ASCII.

Parameter data must be a string or buffer, otherwise there will be an error:

1

TypeError: Not a string or buffer

One liner to create a Node.js HTTP server:

1

require('http').createServer(function (req, res) { res.end('OK'); }).listen(3000);

Print out the process ID instead:

1

require('http').createServer(function (req, res) { res.end('' + process.pid); }).listen(3000);

This is useful with the worker process when implementing the cluster module.

From io.js back to Node.js, time to update my installation guide.

Install via Package Manager

Install in Ubuntu directly via apt:

1

$ sudo apt-get install -y nodejs

However, it will not give you the latest version. So, use an alternative method.

For Node.js v0.12:

1
2

$ curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
$ sudo apt-get install -y nodejs

For io.js v3.x:

1
2

$ curl -sL https://deb.nodesource.com/setup_iojs_3.x | sudo bash -
$ sudo apt-get install -y iojs

Install from the Source

Install build tools:

1

$ sudo apt-get install -y build-essential

Clone the Node.js repository from GitHub:

1

$ git clone https://github.com/nodejs/node.git

Future Node.js releases will be from this repo: https://github.com/nodejs/node.

Check out a specific version:

1

$ cd node && git checkout v3.2.0

Install globally:

1

$ ./configure && make && sudo make install

LoDash has _.template function:

Creates a compiled template function that can interpolate data properties in “interpolate” delimiters, HTML-escape interpolated data properties in “escape” delimiters, and execute JavaScript in “evaluate” delimiters. Data properties may be accessed as free variables in the template. If a setting object is provided it takes precedence over _.templateSettings values.

The template settings affect the behavior of LoDash, and due to Node’s module caching, this will affect the entire application. This will likely cause unexpected error, especially you are writing a module or a library, and the application that requires the module will also use LoDash but with a different template setting:

1

SyntaxError: Unexpected token =

The rule of thumb: Don’t do it!