Why Using Exact Versions in NPM Package Dependencies

Versions frequently used in NPM package dependencies are caret ranges or ^version. And npm install uses caret ranges as well.

Caret range:

Allows changes that do not modify the left-most non-zero digit in the [major, minor, patch] tuple. In other words, this allows patch and minor updates for versions 1.0.0 and above, patch updates for version 0.x >=0.1.0, and no updates for version 0.0.x.

Most of time, using caret ranges as dependency versions work perfectly fine. But once a while, there are bugs. In one of projects, JS-YAML with caret version ^3.4.3 was used and installed:

1
2
3
4
5
{
"dependencies": {
"js-yaml": "^3.4.3"
}
}

Time passes. When npm install was run again from a freshly cloned project code base, version 3.5.2 was installed. Since JS-YAML version 3.5.0, errors are thrown when safe loading a spec with duplicate keys. If there are no duplicate keys in the YAML file, this will be fine. However, one of the files has it. The correct approach is to actually fix the duplicate keys. But it requires additional work back and forth. You’ve heard this before: “Hey it works when I installed it.”

The point here is to use exact versions instead of letting the package manager decides by dropping the caret character:

1
2
3
4
5
{
"dependencies": {
"js-yaml": "3.4.3"
}
}

This will avoid the above mentioned problem. We can manually update the outdated NPM packages.

Don’t let machine decide. You Do!

Setting Up Local DNS Server for Multi-Devices

As always, the first thing you have to ask yourself is that why do you want to set up a DNS server? Here are a few of mine:

  • Access web services via custom domains instead of IP addresses
  • Update multiple devices (including mobile devices) is time consuming and inconvenient
  • Router does not provide a DNS server

I run many web services in the local network, for example, GitLab for code repository and Ghost for blogging. Instead of typing IP address for each service, it will be much easier to use custom domain names, for example, accessing my blog via http://ghost/, not dot com nor localhost.

This can be easily done if there is just one machine. You can update the host configuration file in the local machine such as /etc/hosts in Linux or C:\Windows\System32\drivers\etc\hosts in Windows. But I have multiple devices: a laptop, a tablet, and of course a smartphone. Updating multiple devices is a pain. And for many mobile devices running systems such as iOS or Android, it is not easy to edit the host configuration file without rooting the devices. Therefore, we will opt to update a single DNS configuration file.

DNS configuration can be inherited from the DHCP server from the local network router. Unfortunately, the router I have does not provide a built-in DNS server. I have to setup my own DNS server. Once the DNS server has been setup, the router will use the server as the primary DNS server, and falls back to default gateway IP as the secondary DNS server, or we can use either Google’s public DNS server with IP address 8.8.8.8 or any other alternatives.

  • Primary DNS server: 192.168.0.100
  • Secondary DNS server: 192.168.0.1, 8.8.8.8 or others

I am going to setup a DNS server in a Debian Jessie machine. Here is a summary of steps:

  1. Install BIND
  2. Add custom zone

Keeping Routing Simple and Separated with Koa Router

Debugging an error when using koa-router module that multiple routes were invoked on a single HTTP request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
var app = require('koa')();
var router = require('koa-router')();
router.use(function *(next) {
this.body = {};
yield next;
});
router.get('/users', function *(next) {
this.body['/users'] = true;
yield next;
});
router.get('/users/:id?', function *(next) {
this.body['/users/:id?'] = true;
yield next;
});
app.use(router.routes());
app.listen(3000);

When making a request to get a list of users, it will match both routes:

1
2
3
4
5
6
7
8
9
10
$ http :3000/users
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 34
Content-Type: application/json; charset=utf-8
{
"/users": true,
"/users/:id?": true
}

Because in the second route :id is optional, and since there is yield next statement in the first route, the second route will be executed subsequently.

The cause of the bug is the design of the routing GET /users/:id?. The correct routes are:

  • GET /users: Retrieve a list (array) of users.
  • GET /users/:id: Retrieve a single user object.

The question mark should not be appended to the end of id parameter, which effectively marking the route to perform both duties. It’s better to keep them simple and keep them separated.

Passing Information Through Koa Middleware Via "this.state"

Koa has a context object created during every request/response cycle. The context is referenced via this in the middleware. Much information are captured in the context, such as req and res. In order to avoid polluting the context and colliding with other middleware, the information from your application should be encapsulated into this.state property, a namespace for passing information through middleware during request/response cycle:

1
2
3
4
5
6
7
8
9
10
11
12
require('koa')().use(function *(next) {
var state = this.state;
state.user = yield users.create(this.request.body);
state.code = 201;
state.body = {
name: user.name,
mail: user.mail
};
yield next;
}).listen(3000);

The this.state has been initialized into an empty object:

1
2
3
require('koa')().use(function *(next) {
this.body = this.state;
}).listen(3000);

Therefore, we can start using it right away without doing our own initialization:

1
2
3
4
5
6
7
$ http :3000
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 2
Content-Type: application/json; charset=utf-8
{}

Overriding the Default Forwarded SSH Port in Vagrant

TLDR:

1
2
# Must specified `id: "ssh"` in order to override the default forwarded SSH port.
config.vm.network :forwarded_port, guest: 22, host: 2322, id: "ssh"

What’s the problem? If running multiple virtual machines managed by Vagrant, SSH port collision will happen:

1
2
3
4
5
6
7
8
9
10
11
12
13
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 2222 is already in use
on the host machine.
To fix this, modify your current projects Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:
config.vm.network :forwarded_port, guest: 22, host: 1234
Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding.

The example is correct but incomplete. By default the SSH port (22) on the guest machine is forwarded to port 2222 on the host machine:

1
2
3
4
5
6
7
8
9
# Source: https://github.com/mitchellh/vagrant/blob/v1.7.2/plugins/kernel_v2/config/vm.rb
if !@__networks["forwarded_port-ssh"]
network :forwarded_port,
guest: 22,
host: 2222,
host_ip: "127.0.0.1",
id: "ssh",
auto_correct: true
end

Changing to another port in Vagrantfile does not solve the problem. For example, switching from 2222 to 2322:

1
2
3
4
# Create a forwarded port mapping which allows access to a specific port
# within the machine from a port on the host machine. In the example below,
# accessing "localhost:8080" will access port 80 on the guest machine.
config.vm.network "forwarded_port", guest: 22, host: 2322

In this situation, you are forwarding both ports 2222 and 2322 to the SSH port on the guest machine, as shown in VirtualBox port forwarding settings:

Port Forwarding Rules

When bringing up the machine, there will be multiple host machine ports forwarding to the same SSH port.

1
2
3
4
$ vagrant up
==> default: Forwarding ports...
default: 22 => 2322 (adapter 1)
default: 22 => 2222 (adapter 1)

The SSH configuration shows the same outcome:

1
2
3
4
5
$ vagrant ssh-config
Host default
HostName 127.0.0.1
User chao
Port 2222

Of course, you can do:

1
$ ssh -p 2322 chao@localhost

This works, but we like the simple vagrant ssh command, and we just want to retain a single port 2322.

The solution is appending id: "ssh", and it must be specified in order to override the default forwarded SSH port. See < https://github.com/mitchellh/vagrant/issues/3232#issuecomment-39354659>:

1
2
# Must specified `id: "ssh"` in order to override the default forwarded SSH port.
config.vm.network :forwarded_port, guest: 22, host: 2322, id: "ssh"

After appending id: "ssh", the SSH configuration has also been updated:

1
2
3
4
5
$ vagrant ssh-config
Host default
HostName 127.0.0.1
User chao
Port 2322

Now, you are good to go:

1
$ vagrant ssh

Re-assembling the Requested URL

It is simple to know the request URL path, it’s one (this.url in Koa) used by routing, but if you are looking to get the complete URL, it’s a little bit complicated. Luckily, there is a quick way to re-assemble the requested URL in Koa, if proxy is not a concern:

1
var url = this.request.protocol + '://' + this.request.host + this.request.originalUrl;

Or a shorter version:

1
var url = this.protocol + '://' + this.host + this.originalUrl;

The port number is included in this.host, for example: localhost:3000.

Putting into a complete script:

1
2
3
4
require('koa')().use(function *() {
// Echo back the request URL.
this.body = this.protocol + '://' + this.host + this.originalUrl;
}).listen(3000);

Assume that you have a server running on the host www.example.com with SSL enabled. If you make a request to the server with the URL: https://www.example.com/path, this URL will be echoed back in the response by the above script.

Calculate Hashes for All OpenSSL Digest Algorithms

Based on the following Node.js statement:

1
require('crypto').createHash(algorithm).update('secret', 'utf8').digest('base64');

On a Ubuntu 14.04 system with OpenSSL:

1
2
$ openssl version
OpenSSL 1.0.1f 6 Jan 2014

List available digest algorithms:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
$ openssl list-message-digest-algorithms
DSA
DSA-SHA
DSA-SHA1 => DSA
DSA-SHA1-old => DSA-SHA1
DSS1 => DSA-SHA1
MD4
MD5
RIPEMD160
RSA-MD4 => MD4
RSA-MD5 => MD5
RSA-RIPEMD160 => RIPEMD160
RSA-SHA => SHA
RSA-SHA1 => SHA1
RSA-SHA1-2 => RSA-SHA1
RSA-SHA224 => SHA224
RSA-SHA256 => SHA256
RSA-SHA384 => SHA384
RSA-SHA512 => SHA512
SHA
SHA1
SHA224
SHA256
SHA384
SHA512
DSA
DSA-SHA
dsaWithSHA1 => DSA
dss1 => DSA-SHA1
ecdsa-with-SHA1
MD4
md4WithRSAEncryption => MD4
MD5
md5WithRSAEncryption => MD5
ripemd => RIPEMD160
RIPEMD160
ripemd160WithRSA => RIPEMD160
rmd160 => RIPEMD160
SHA
SHA1
sha1WithRSAEncryption => SHA1
SHA224
sha224WithRSAEncryption => SHA224
SHA256
sha256WithRSAEncryption => SHA256
SHA384
sha384WithRSAEncryption => SHA384
SHA512
sha512WithRSAEncryption => SHA512
shaWithRSAEncryption => SHA
ssl2-md5 => MD5
ssl3-md5 => MD5
ssl3-sha1 => SHA1
whirlpool

Calculate hashes for each and every uniquely listed algorithm:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
$ openssl list-message-digest-algorithms | \
cut -d ' ' -f 1 | \
xargs -n 1 -I {} node -e "algorithm = '{}'; \
digest = require('crypto').createHash('{}').update('secret', 'utf8').digest('base64'); \
console.log(algorithm, digest)" | \
sort -u
DSA 5en6G6MezRroT3XKqkdPOmY/BfQ=
DSA-SHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
DSA-SHA1-old 5en6G6MezRroT3XKqkdPOmY/BfQ=
DSA-SHA 5en6G6MezRroT3XKqkdPOmY/BfQ=
dsaWithSHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
dss1 5en6G6MezRroT3XKqkdPOmY/BfQ=
DSS1 5en6G6MezRroT3XKqkdPOmY/BfQ=
ecdsa-with-SHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
md4WithRSAEncryption Z9Pa/vY/8AYDru83ac+/DQ==
MD4 Z9Pa/vY/8AYDru83ac+/DQ==
md5WithRSAEncryption Xr4ilOzQ4PCOq3aQ0qbuaQ==
MD5 Xr4ilOzQ4PCOq3aQ0qbuaQ==
ripemd160WithRSA zZi/AgLvB+OOh/a9lEXl5zMeLHg=
RIPEMD160 zZi/AgLvB+OOh/a9lEXl5zMeLHg=
ripemd zZi/AgLvB+OOh/a9lEXl5zMeLHg=
rmd160 zZi/AgLvB+OOh/a9lEXl5zMeLHg=
RSA-MD4 Z9Pa/vY/8AYDru83ac+/DQ==
RSA-MD5 Xr4ilOzQ4PCOq3aQ0qbuaQ==
RSA-RIPEMD160 zZi/AgLvB+OOh/a9lEXl5zMeLHg=
RSA-SHA1-2 5en6G6MezRroT3XKqkdPOmY/BfQ=
RSA-SHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
RSA-SHA224 lcf7ypKsUIOv2mKlZKPQFPw7cskUDjy5nqa/Eg==
RSA-SHA256 K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
RSA-SHA384 WKd1ukESvjAFrkQHznV9iP2nHUBJe7gCbsrFTU4//HIyzo3jq1rLMK45dg/ufFPt
RSA-SHA512 vSsar3708Jvp9Szi2NWZZ02Bqp1qRCFpbcTZPdBhnWgs5WtNZKnvCXdhztmeD2cmW192CF5bDufKRpayrW/isg==
RSA-SHA f72qWouzjoc6vqk8vf5NBIVHFXs=
SHA1 5en6G6MezRroT3XKqkdPOmY/BfQ=
sha1WithRSAEncryption 5en6G6MezRroT3XKqkdPOmY/BfQ=
SHA224 lcf7ypKsUIOv2mKlZKPQFPw7cskUDjy5nqa/Eg==
sha224WithRSAEncryption lcf7ypKsUIOv2mKlZKPQFPw7cskUDjy5nqa/Eg==
SHA256 K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
sha256WithRSAEncryption K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols=
sha384WithRSAEncryption WKd1ukESvjAFrkQHznV9iP2nHUBJe7gCbsrFTU4//HIyzo3jq1rLMK45dg/ufFPt
SHA384 WKd1ukESvjAFrkQHznV9iP2nHUBJe7gCbsrFTU4//HIyzo3jq1rLMK45dg/ufFPt
SHA512 vSsar3708Jvp9Szi2NWZZ02Bqp1qRCFpbcTZPdBhnWgs5WtNZKnvCXdhztmeD2cmW192CF5bDufKRpayrW/isg==
sha512WithRSAEncryption vSsar3708Jvp9Szi2NWZZ02Bqp1qRCFpbcTZPdBhnWgs5WtNZKnvCXdhztmeD2cmW192CF5bDufKRpayrW/isg==
SHA f72qWouzjoc6vqk8vf5NBIVHFXs=
shaWithRSAEncryption f72qWouzjoc6vqk8vf5NBIVHFXs=
ssl2-md5 Xr4ilOzQ4PCOq3aQ0qbuaQ==
ssl3-md5 Xr4ilOzQ4PCOq3aQ0qbuaQ==
ssl3-sha1 5en6G6MezRroT3XKqkdPOmY/BfQ=
whirlpool 4GG4emdK44gOFZq1XtNdbF6Kau+sarCDBKUFiAGNN3soY5uxX97t8AbVfkX3tCmObf786vfJLIJqcI/m0RVusw==

Get Started with the Cut Command of GNU Coreutils

A part of GNU Coreutils, cut command likes paste and join is one that operates on fields. The cut command prints selected parts (fields or sections) of lines.

Things to keep in mind:

  • Work on text stream line by line
  • Print sections/parts/fields
  • Either -c, -b or -f option must be used
  • The default delimiter is TAB
  • The delimiter must be a single character
  • Consecutive delimiters need to be consolidated into one
  • Cut multiple or a range of fields

Generate sample data and saved in the file tab.txt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ file=tab.txt && \
for i in $(seq 1 9); do \
for j in $(seq 1 9); do \
if [ "$j" == "9" ]; then \
echo -e "$i$j"; \
else \
echo -en "$i$j\t"; \
fi; \
done; \
done > $file && cat $file
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

Fields on each line is separated by a tab character.

There is a required option:

1
2
3
$ cut tab.txt
cut: you must specify a list of bytes, characters, or fields
Try 'cut --help' for more information.

These options are:

  • bytes: -b or --bytes=LIST
  • characters: -c or --characters=LIST
  • fields: -f or --fields=LIST

This is very odd. Option means optional, a default should be provided. But let’s focus on the common used one: fields.

Cut the first and the ninth fields with the default delimiter TAB:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ cut -f 1 tab.txt
11
21
31
41
51
61
71
81
91
$ cut -f 9 tab.txt
19
29
39
49
59
69
79
89
99

Use space as the delimiter:

1
2
3
4
5
6
7
8
9
10
$ cp tab.txt space.txt && sed -i 's/\t/ /g' $_ && cat $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

We must choose a different delimiter via -d or --delimiter=DELIM option:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ cut -f 1 $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99
$ cut -f 1 -d ' ' $_
11
21
31
41
51
61
71
81
91

Delimiter must be a single character:

1
2
3
$ cut -f 1 -d '\s' $_
cut: the delimiter must be a single character
Try 'cut --help' for more information.

Files containing mixed delimiters (tab and space):

1
2
3
4
5
6
7
8
9
10
$ cp tab.txt mixed.txt && sed -i 's/\(9.\)\t/\1 /g' $_ && cat $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

will print any line that contains no delimiter character:

1
2
3
4
5
6
7
8
9
10
$ cut -f 1 mixed.txt
11
21
31
41
51
61
71
81
91 92 93 94 95 96 97 98 99

Or use -s or --only-delimited option to omit those lines.

1
2
3
4
5
6
7
8
9
$ cut -sf 1 mixed.txt
11
21
31
41
51
61
71
81

But the better approach is to do data cleansing prior.

What about multiple TAB characters in the file:

1
2
3
4
5
6
7
8
9
10
$ sed -i 's/\(1.\)\t/\1\t\t/' mixed.txt && cat $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

An empty field is still a field:

1
2
3
4
5
6
7
8
9
$ cut -sf 2 mixed.txt
22
32
42
52
62
72
82

Therefore, the drawback here is that there cannot be multiple delimiter sticking together. Must perform data cleansing to reduce consecutive delimiters into a single one:

1
2
3
4
5
6
7
8
9
10
$ sed -i 's/\t\+/\t/g' mixed.txt && cat $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

Multiple fields can be cut:

1
2
3
4
5
6
7
8
9
10
$ cut -f 1,3,5,7,9 tab.txt
11 13 15 17 19
21 23 25 27 29
31 33 35 37 39
41 43 45 47 49
51 53 55 57 59
61 63 65 67 69
71 73 75 77 79
81 83 85 87 89
91 93 95 97 99

Cut a range:

1
2
3
4
5
6
7
8
9
10
$ cut -f 3-5 tab.txt
13 14 15
23 24 25
33 34 35
43 44 45
53 54 55
63 64 65
73 74 75
83 84 85
93 94 95

Cut up to or from a field:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ cut -f -3 tab.txt
11 12 13
21 22 23
31 32 33
41 42 43
51 52 53
61 62 63
71 72 73
81 82 83
91 92 93
$ cut -f 7- tab.txt
17 18 19
27 28 29
37 38 39
47 48 49
57 58 59
67 68 69
77 78 79
87 88 89
97 98 99

When cut multiple fields, the fields are separated by the same delimiter used (indicated by -d field or TAB as the default). If change the output delimiter, it’s not the job of cut, pipe to another program:

1
2
3
4
5
6
7
8
9
10
$ cut -f 3-5 tab.txt | sed 's/\t/ /g'
13 14 15
23 24 25
33 34 35
43 44 45
53 54 55
63 64 65
73 74 75
83 84 85
93 94 95

Node.js Crypto Starters

Starter for the most common usage.

Create a MD5 hex hash:

1
require('crypto').createHash('md5').update(data, 'utf8').digest('hex');

Create a SHA256 base64 hash:

1
require('crypto').createHash('sha256').update(data, 'utf8').digest('base64');

Calculating the digest varies with encoding:

1
2
3
4
5
6
7
8
9
10
11
12
> data = 'secret'
'secret'
> require('crypto').createHash('sha256').update(data, 'utf8').digest('hex');
'2bb80d537b1da3e38bd30361aa855686bde0eacd7162fef6a25fe97bf527a25b'
> require('crypto').createHash('sha256').update(data, 'utf8').digest('base64');
'K7gNU3sdo+OL0wNhqoVWhr3g6s1xYv72ol/pe/Unols='
> require('crypto').createHash('sha256').update(data, 'utf8').digest('binary');
'+¸\rS{\u001d£ãÓ\u0003aªV½àêÍqbþö¢_é{õ\'¢['
> require('crypto').createHash('sha256').update(data, 'utf8').digest();
<Buffer 2b b8 0d 53 7b 1d a3 e3 8b d3 03 61 aa 85 56 86 bd e0 ea cd 71 62 fe f6 a2 5f e9 7b f5 27 a2 5b>
> require('crypto').createHash('sha256').update(data, 'utf8').digest().toString();
'+�\rS{\u001d����\u0003a��V�����qb���_�{�\'�['

Both hex and base64 are commonly used ones.

Needs an input_encoding when update data, otherwise there will be different results:

1
2
3
4
5
6
7
8
9
10
> data = '秘密'
'秘密'
> require('crypto').createHash('sha256').update(data, 'ascii').digest('hex');
'7588bc51fd0ff10db9c66549e5cb7969b9f6b7cf3ccee080137a8eb1aa06e718'
> require('crypto').createHash('sha256').update(data, 'utf8').digest('hex');
'062a2931da683a9897c5a0b597113b1e9fd0d5bfb63e2a5d7c88b724f7f55c02'
> require('crypto').createHash('sha256').update(data, 'binary').digest('hex');
'7588bc51fd0ff10db9c66549e5cb7969b9f6b7cf3ccee080137a8eb1aa06e718'
> require('crypto').createHash('sha256').update(data).digest('hex');
'7588bc51fd0ff10db9c66549e5cb7969b9f6b7cf3ccee080137a8eb1aa06e718'

Don’t assume ASCII.

Parameter data must be a string or buffer, otherwise there will be an error:

1
TypeError: Not a string or buffer

Add a Custom Domain to the Specific Module of Google App Engine Managed VMs

I would like to add a custom domain to a module other than the default one in Google App Engine Managed VMs. For example, I have the following sub-domains:

1
2
3
4
Custom domain names | SSL support | Record type | Data | Alias
------------------- | ----------- | ----------- | -------------------- | -----
api.example.com | none | CNAME | ghs.googlehosted.com | api
admin.example.com | none | CNAME | ghs.googlehosted.com | admin

listed in the application console:

https://console.developers.google.com/project/[project]/appengine/settings/domains

The domain api.example.com points to the default module. And I would like the other domain admin.example.com to point to the module admin. However, by merely adding the custom domain in the settings, both api.example.com and admin.example.com are pointed to the same module: the default module. Then how to point the custom domain and route the traffic to the admin module? The answer is on the dispatch file. But first need to understand the concept of modules in Google App Engine.

Google App Engine Module Hierarchy
Source: Google App Engine Docs

The above chart illustrates the architecture of a Google App Engine application:

  • Application: “An App Engine application is made up of one or more modules”. [1]
  • Module: “Each module consists of source code and configuration files. The files used by a module represents a version of the module. When you deploy a module, you always deploy a specific version of the module.” [1] “All module-less apps have been converted to contain a single default module.” [1] Every application has a single default module.
  • Version: “A particular module/version will have one or more instances.”
  • Instance: “Each instance runs its own separate executable.”

Another concept is the resource sharing:

  • “App Engine Modules let developers factor large applications into logical components that can share stateful services and communicate in a secure fashion.” [1]
  • “Stateful services (such as Memcache, Datastore, and Task Queues) are shared by all modules in an application.” [1]

Remember that Google Cloud Platform is project based. If there are a web application and a mobile application, and they are both making requests to another application, the API. Then there should be three projects. However, in this case, both api and admin share the same services, such as the same database and file storage. We should put them together in the same application as separate modules.

With that in mind, how to route requests to a specific module? “Every module, version, and instance has its own unique URI (for example, v1.my-module.my-app.appspot.com). Incoming user requests are routed to an instance of a particular module/version according to URL addressing conventions and an optional customized dispatch file.” [1] Since a custom domain is used, we have to use the customized dispatch file. And this is done by creating a dispatch file dispatch.yml to route requests based on URL patterns.

1
2
3
4
5
6
7
8
# Dispatch
# ========
---
dispatch:
- url: '*/favicon.ico'
module: default
- url: 'admin.example.com/*'
module: admin

The application field is not necessary, otherwise, you will get:

1
2
WARNING: The [application] field is specified in file [/home/chao/example/dispatch.yml].
This field is not used by gcloud and should be removed.

Glob characters (such as asterisk) can be used, and support up to 10 routing rules.

Deploy the dispatch file is simple:

1
$ gcloud preview app deploy dispatch.yml

And it should require almost no time to wait. Once it is ready, the requests sent to admin.example.com will be routed properly to the admin module.