cURL: Read from File or Stdin

Redirect or pipe the content of a file or the output of another command as the input of cURL, for example, making a POST request with cURL.

This is done by prefixing the data parameter with @, the rest is the filename to read data from. With -, the data is coming from stdin:

1
2
3
4
5
6
7
8
$ seq 17000 1 17001 | \
curl \
--request POST \
--include \
--header 'Content-Type: application/json' \
--data-binary @- \
--no-buffer \
https://example.com/data

Command seq 17000 1 17001 returns:

1
2
17000
17001

The size of the above two numbers is 12 bytes (including two newlines). But when reading data via @ with --data option, “carriage returns and newlines will be stripped out” [1]. If you check your server, you will see the content length is only 10 bytes.

To correct this problem, we will send the raw data using --data-binary option instead. This preserves \n and \r characters.

Changelog Starter

A changelog starter to be consistent with my coding style:

  • Two header levels only with setext style.
  • Use underlines for first-level headers and dashes for second-level headers.
  • Each release is a section, hence use second-level header.
  • Each category in a release is not a section, hence use emphasis with double asterisks.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
Changelog
=========
Guideline:
1. Record notable changes.
2. List releases in reversed chronological order, newest on top.
3. Write all dates in `YYYY-MM-DD` format.
4. Follow [SemVer] format for each release.
5. Creating a new release by copying the **Example** section (on the bottom)
to the top and rename it **Unreleased**.
6. Keep an **Unreleased** section on top for tracking any changes in order to
minimize required effort.
7. Change _Unreleased_ to a release name with the date when ready.
_The guideline attempts to follow [Keep a CHANGELOG][1]._
[SemVer]: http://semver.org/
[1]: http://keepachangelog.com/
0.1.0 / 2015-01-01
------------------
**Security**
- Inform users to upgrade for vulnerabilities.
**Removed**
- Remove a deprecated feature in this release.
**Deprecated**
- Deprecate a feature in the upcoming release.
**Changed**
- Change an existing functionality.
**Added**
- Add a new feature.
**Fixed**
- Fix a bug.
Example / YYYY-MM-DD
--------------------
**Security**
- Inform users to upgrade for vulnerabilities.
- Each sentence must use the present tense and ends with a period.
**Removed**
- Remove a deprecated feature in this release.
- Basic Authentication is dropped in favor of Digest Authentication.
**Deprecated**
- Deprecate a feature in the upcoming release.
- Basic Authentication will be replaced by Digest Authentication in the next
release.
**Changed**
- Change an existing functionality.
- User session data is added to the user object upon initialization.
**Added**
- Add a new feature.
- A new endpoint `/users` is added to handle user CRUD.
**Fixed**
- Fix a bug.
- Add the default case to handle unknown cases in user group assigning.

Rules on Structuring the Test Directory Structure with Mocha

When writing tests, the framework I use for Node.js is Mocha (reviewed version: v2.2.x). But how should I structure the ./test directory? I first started by grouping related tests into directory, for example:

1
2
3
4
5
6
$ tree ./test
./test
├── posts
│ └── main.js
└── users
└── main.js

You can use mocha --recursive to run all test suites.

However, it’s unlikely that all files in the test directory are test suites. You might need to define some shared utilities, data generation scripts, or even setup files. You don’t want these files to be executed by Mocha. You can get away by using:

  • mocha --grep or mocha --fgrep
  • Use find and xargs to locate the files to be executed by Mocha

But these are getting more and more complex. You want something simple and dumb frankly. Therefore, a better approach is to follow the two guidelines. One is the default behavior of Mocha:

Run LoDash in Context to Change Template Settings

LoDash has _.template function:

Creates a compiled template function that can interpolate data properties in “interpolate” delimiters, HTML-escape interpolated data properties in “escape” delimiters, and execute JavaScript in “evaluate” delimiters. Data properties may be accessed as free variables in the template. If a setting object is provided it takes precedence over _.templateSettings values.

The template settings affect the behavior of LoDash, and due to Node’s module caching, this will affect the entire application. This will likely cause unexpected error, especially you are writing a module or a library, and the application that requires the module will also use LoDash but with a different template setting:

1
SyntaxError: Unexpected token =

The rule of thumb: Don’t do it!

Use Custom Message in Chai

When running a test in a loop with Mocha and Chai, it might not be easy to pinpoint the element where the error occurred. For example:

1
2
3
for (let i = 0, len = arr.length; i < len; i += 1) {
expect(arr[i]).to.equal(true);
}

Run the test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ mocha --harmony test/message.js
Custom Message
1) Should return true
0 passing (37ms)
1 failing
1) Custom Message Should return true:
AssertionError: expected false to equal true
+ expected - actual
-false
+true
at Context.<anonymous> (test/message.js:16:25)

It failed on the expect statement, but the error message does not show which element is in error.

Instead, we can add a custom message to the expect statement:

1
2
3
for (let i = 0, len = arr.length; i < len; i += 1) {
expect(arr[i], 'i = ' + i).to.equal(true);
}

Fail to Upload Large File to Google App Engine Managed VMs

After deploying a Node.js app with custom runtime to Google App Engine, when uploading file larger than 1MB, the server returns HTTP 413 status code, meaning the server is refusing to process the request because the uploading entity size is too large.

I had the following setup:

  • The App Engine application is Google managed
  • The application is Node.js app with custom Dockerfile
  • The application Docker container runs node command directly on port 8080

After accessing the server (which changed from Google managed to user managed), and checked the Docker containers:

1
2
$ sudo docker ps -a
gcr.io/google_appengine/nginx-proxy:latest "/usr/sbin/nginx"

It looks like Nginx is running as the proxy to Node.js application. When uploading file that is larger than 1MB, the response is HTTP 413 error:

1
2
3
4
5
6
7
8
9
10
11
12
<html>
<head>
<title>413 Request Entity Too Large</title>
</head>
<body bgcolor="white">
<center>
<h1>413 Request Entity Too Large</h1>
</center>
<hr>
<center>nginx</center>
</body>
</html>

And the Nginx access log from the server:

1
2
$ tail /var/log/nginx/error.log
2015/06/25 12:00:00 [error] 7#0: *101623 client intended to send too large body: 1696805 bytes

By default, Nginx limits uploading size or client_max_body_size to be 1MB. The default size is just too small. A short video upload could easily exceed the limit. So, how to accept file upload larger than 1MB via Google Managed VMs in App Engine? Is there a way to change the value through app.yamlcustom runtime configuration file? Or a custom Nginx Docker container can be deployed along with the application Docker container?

Migrate to Cloud9 IDE

My blog is powered by Hexo:

Hexo is a fast, simple and powerful blog framework. You write posts in Markdown (or other languages) and Hexo generates static files with a beautiful theme in seconds. - https://hexo.io/docs/

To generate static files, I need a Node.js development environment. But as I move around from different computers and operating systems, this becomes inconvenient. I need a cloud development environment where I can access it from any device.

So, I have decided to move my blog to Cloud9 IDE, a development environment in the cloud. And this is the first blog from the environment.

Looking forward!