Setup Development Environment with Vagrant on Google Compute Engine

We usually use Vagrant to provision and manage VirtualBox virtual machines or VMs. And Vagrant only ships with VirtualBox support by default. But Vagrant can do much more. VirtualBox is just one of providers, additional providers can be added via the Vagrant plugin system. Here, we are going to use Google Compute Engine as the provider and setup a disposable development environment.

Basic steps involve:

  1. Install Vagrant
  2. Install vagrant-google plugin
  3. Add SSH keys to metadata server
  4. Add Vagrant box
  5. Add a Vagrantfile and override the defaults from the box
  6. Provision the machine

First, install Vagrant (exampled by Debian systems such as Ubuntu):

1
2
3
4
5
$ file=/tmp/vagrant.deb version=1.7.4 && \
curl -sSL https://dl.bintray.com/mitchellh/vagrant/vagrant_${version}_x86_64.deb > ${file} && \
sudo dpkg -i ${file} && \
rm ${file} && \
vagrant version

Install dependencies for building Debian packages, which are needed for the plugin:

1
$ sudo apt-get install -y build-essential

Install vagrant-google plugin:

1
2
3
$ vagrant plugin install vagrant-google
Installing the 'vagrant-google' plugin. This can take a few minutes...
Installed the plugin 'vagrant-google (0.2.1)'!

In order to SSH into the VM, we need to add public key to the instance which will be provisioned. In Google Compute Engine, this is done via metadata server at the project level.

Add the username to prefix the SSH key, so the Compute Engine will pick it up and create the corresponding user:

1
2
$ echo -n 'chao:' > /tmp/id_rsa.pub
$ cat ~/.ssh/id_rsa.pub >> /tmp/id_rsa.pub

Now the concatenated one will have the username:

1
chao:ssh-rsa AAAAB3... chao@ubuntu

Setting up SSH keys at the project level:

1
2
$ gcloud compute project-info add-metadata --metadata-from-file \
sshKeys=/tmp/id_rsa.pub

Google Cloud SDK or gcloud command should be installed already. Now it’s time to configure the machine to be provisioned.

The plugin provides a Vagrant box, which is a package format for Vagrant environments. It is not necessary to create a new box, we can just override the defaults by the Vagrantfile. So add the provider-specific box:

1
$ vagrant box add gce https://github.com/mitchellh/vagrant-google/raw/master/google.box

Make sure the box is added:

1
$ vagrant box list

Add a Vagrantfile and override the defaults:

Upgrade Vagrant in Windows from Command Line without Prompt

To determine whether your Vagrant is out of date or not, issue:

1
2
3
4
5
$ vagrant version
Installed Version: 1.7.4
Latest Version: 1.7.4
You're running an up-to-date version of Vagrant!

If yours is out of date, you can download it from its download page. However, being a lazy programmer, that’s still too many clicks. I want to figure out a way to update or install Vagrant from the command line without prompt.

And here you go!

Vagrant binaries are distributed by Bintray, its format is:

1
https://dl.bintray.com/mitchellh/vagrant/vagrant_{version}.msi

For example:

1
https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.4.msi

After you issue vagrant version, you know the correct version to upgrade, then download it:

1
$ curl -sSL https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.4.msi > vagrant.msi

Install it by msiexec:

1
$ msiexec /i C:\Users\Chao\Downloads\vagrant.msi /passive /norestart

Make sure to use the full path instead of relative .\vagrant.msi, and Vagrant should be installed in the default location C:\HashiCorp\Vagrant. To find out all possible options, Just type msiexec command only.

Saving a few clicks!

Install Docker on Google Cloud Platform with Ubuntu Vivid 15.04

Ubuntu 15.04 (Vivid Vervet) has switched the init manager from Upstart to systemd. Will the Docker installation change? Let’s give a try and install it on a Google Compute Engine instance.

Install Docker:

1
$ curl -sSL https://get.docker.com/ | sh

Add user to Docker group for running commands without sudo:

1
$ sudo usermod -aG docker ${USER}

Log out, then log back in.

Verify the installation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ docker run hello-world
Hello from Docker.
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker Hub account:
https://hub.docker.com
For more examples and ideas, visit:
https://docs.docker.com/userguide/

For upgrading, the process is the same as installing:

1
$ curl -sSL https://get.docker.com/ | sh

Node.js HTTP Server One Liner

One liner to create a Node.js HTTP server:

1
require('http').createServer(function (req, res) { res.end('OK'); }).listen(3000);

Print out the process ID instead:

1
require('http').createServer(function (req, res) { res.end('' + process.pid); }).listen(3000);

This is useful with the worker process when implementing the cluster module.

Various Node.js Installation Methods

From io.js back to Node.js, time to update my installation guide.

Install via Package Manager

Install in Ubuntu directly via apt:

1
$ sudo apt-get install -y nodejs

However, it will not give you the latest version. So, use an alternative method.

For Node.js v0.12:

1
2
$ curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
$ sudo apt-get install -y nodejs

For io.js v3.x:

1
2
$ curl -sL https://deb.nodesource.com/setup_iojs_3.x | sudo bash -
$ sudo apt-get install -y iojs

Install from the Source

Install build tools:

1
$ sudo apt-get install -y build-essential

Clone the Node.js repository from GitHub:

1
$ git clone https://github.com/nodejs/node.git

Future Node.js releases will be from this repo: https://github.com/nodejs/node.

Check out a specific version:

1
$ cd node && git checkout v3.2.0

Install globally:

1
$ ./configure && make && sudo make install

cURL: Read from File or Stdin

Redirect or pipe the content of a file or the output of another command as the input of cURL, for example, making a POST request with cURL.

This is done by prefixing the data parameter with @, the rest is the filename to read data from. With -, the data is coming from stdin:

1
2
3
4
5
6
7
8
$ seq 17000 1 17001 | \
curl \
--request POST \
--include \
--header 'Content-Type: application/json' \
--data-binary @- \
--no-buffer \
https://example.com/data

Command seq 17000 1 17001 returns:

1
2
17000
17001

The size of the above two numbers is 12 bytes (including two newlines). But when reading data via @ with --data option, “carriage returns and newlines will be stripped out” [1]. If you check your server, you will see the content length is only 10 bytes.

To correct this problem, we will send the raw data using --data-binary option instead. This preserves \n and \r characters.

Changelog Starter

A changelog starter to be consistent with my coding style:

  • Two header levels only with setext style.
  • Use underlines for first-level headers and dashes for second-level headers.
  • Each release is a section, hence use second-level header.
  • Each category in a release is not a section, hence use emphasis with double asterisks.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
Changelog
=========
Guideline:
1. Record notable changes.
2. List releases in reversed chronological order, newest on top.
3. Write all dates in `YYYY-MM-DD` format.
4. Follow [SemVer] format for each release.
5. Creating a new release by copying the **Example** section (on the bottom)
to the top and rename it **Unreleased**.
6. Keep an **Unreleased** section on top for tracking any changes in order to
minimize required effort.
7. Change _Unreleased_ to a release name with the date when ready.
_The guideline attempts to follow [Keep a CHANGELOG][1]._
[SemVer]: http://semver.org/
[1]: http://keepachangelog.com/
0.1.0 / 2015-01-01
------------------
**Security**
- Inform users to upgrade for vulnerabilities.
**Removed**
- Remove a deprecated feature in this release.
**Deprecated**
- Deprecate a feature in the upcoming release.
**Changed**
- Change an existing functionality.
**Added**
- Add a new feature.
**Fixed**
- Fix a bug.
Example / YYYY-MM-DD
--------------------
**Security**
- Inform users to upgrade for vulnerabilities.
- Each sentence must use the present tense and ends with a period.
**Removed**
- Remove a deprecated feature in this release.
- Basic Authentication is dropped in favor of Digest Authentication.
**Deprecated**
- Deprecate a feature in the upcoming release.
- Basic Authentication will be replaced by Digest Authentication in the next
release.
**Changed**
- Change an existing functionality.
- User session data is added to the user object upon initialization.
**Added**
- Add a new feature.
- A new endpoint `/users` is added to handle user CRUD.
**Fixed**
- Fix a bug.
- Add the default case to handle unknown cases in user group assigning.

Rules on Structuring the Test Directory Structure with Mocha

When writing tests, the framework I use for Node.js is Mocha (reviewed version: v2.2.x). But how should I structure the ./test directory? I first started by grouping related tests into directory, for example:

1
2
3
4
5
6
$ tree ./test
./test
├── posts
│ └── main.js
└── users
└── main.js

You can use mocha --recursive to run all test suites.

However, it’s unlikely that all files in the test directory are test suites. You might need to define some shared utilities, data generation scripts, or even setup files. You don’t want these files to be executed by Mocha. You can get away by using:

  • mocha --grep or mocha --fgrep
  • Use find and xargs to locate the files to be executed by Mocha

But these are getting more and more complex. You want something simple and dumb frankly. Therefore, a better approach is to follow the two guidelines. One is the default behavior of Mocha:

Run LoDash in Context to Change Template Settings

LoDash has _.template function:

Creates a compiled template function that can interpolate data properties in “interpolate” delimiters, HTML-escape interpolated data properties in “escape” delimiters, and execute JavaScript in “evaluate” delimiters. Data properties may be accessed as free variables in the template. If a setting object is provided it takes precedence over _.templateSettings values.

The template settings affect the behavior of LoDash, and due to Node’s module caching, this will affect the entire application. This will likely cause unexpected error, especially you are writing a module or a library, and the application that requires the module will also use LoDash but with a different template setting:

1
SyntaxError: Unexpected token =

The rule of thumb: Don’t do it!

Use Custom Message in Chai

When running a test in a loop with Mocha and Chai, it might not be easy to pinpoint the element where the error occurred. For example:

1
2
3
for (let i = 0, len = arr.length; i < len; i += 1) {
expect(arr[i]).to.equal(true);
}

Run the test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ mocha --harmony test/message.js
Custom Message
1) Should return true
0 passing (37ms)
1 failing
1) Custom Message Should return true:
AssertionError: expected false to equal true
+ expected - actual
-false
+true
at Context.<anonymous> (test/message.js:16:25)

It failed on the expect statement, but the error message does not show which element is in error.

Instead, we can add a custom message to the expect statement:

1
2
3
for (let i = 0, len = arr.length; i < len; i += 1) {
expect(arr[i], 'i = ' + i).to.equal(true);
}