http

Streaming HTTP Request Directly to Response in Node.js

This is a Node.js starting script to stream HTTP request directly into response:

1
2
3
require('http').createServer((req, res) => {
req.pipe(res); // Pipe request directly to response
}).listen(3000);

It behaves almost like an echo, you get back whatever you sent. For example, use HTTPie to make a request to the above server:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ echo foo | http --verbose --stream :3000 Content-Type:text/plain
POST / HTTP/1.1
Accept: application/json, */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 4
Host: localhost:3000
User-Agent: HTTPie/0.9.6
Content-Type: text/plain
foo
HTTP/1.1 200 OK
Connection: keep-alive
Transfer-Encoding: chunked
foo

We can also add the Content-Type response header to echo back what the entire media type is after assembling all chunks.

1
2
3
4
5
6
7
require('http').createServer((req, res) => {
req.pipe(res); // Pipe request directly to response
if (req.headers['content-type']) {
res.setHeader('Content-Type', req.headers['content-type']);
}
}).listen(3000);

The response should have the Content-Type field as below:

1
2
3
4
5
6
HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: text/plain
Transfer-Encoding: chunked
foo

Notice that instead of usual Content-Length in the response header, we’ve got Transfer-Encoding: chunked. The default transfer encoding for Node.js HTTP is chunked:

Sending a ‘Content-length’ header will disable the default chunked encoding.[^1]

About transfer encoding:

Chunked transfer encoding is a data transfer mechanism in version 1.1 of the Hypertext Transfer Protocol (HTTP) in which data is sent in a series of “chunks”. It uses the Transfer-Encoding HTTP header in place of the Content-Length header, which the earlier version of the protocol would otherwise require. Because the Content-Length header is not used, the sender does not need to know the length of the content before it starts transmitting a response to the receiver. Senders can begin transmitting dynamically-generated content before knowing the total size of that content. … The size of each chunk is sent right before the chunk itself so that the receiver can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero.[^2]

With the above starting script, now you can attach some transform streams to manipulate the request and stream back in chunked response.

Settings:

1
2
3
4
$ node --version
v6.3.1
$ http --version
0.9.6

[^1]: HTTP, Node.js API Docs

[^2]: Chunked transfer encoding, Wikipedia

Node.js HTTP Server One Liner

One liner to create a Node.js HTTP server:

1
require('http').createServer(function (req, res) { res.end('OK'); }).listen(3000);

Print out the process ID instead:

1
require('http').createServer(function (req, res) { res.end('' + process.pid); }).listen(3000);

This is useful with the worker process when implementing the cluster module.

SuperTest: Listen at Random Port

In the example section of SuperTest (v0.15.0), it mentions something about ephemeral port:

You may pass an http.Server, or a Function to request() - if the server is not already listening for connections then it is bound to an ephemeral port for you so there is no need to keep track of ports. - https://github.com/visionmedia/supertest#example

Let’s dissect this sentence. There are a few key concepts to grasp:

  1. The request function will accept either a function or an http.Server object.
  2. When does the server not listening for connections?
  3. What is an ephemeral port?

Taking a peek at the index.js file in SuperTest source code, it is easy to see that it accepts both function and http.Server object, but prefer latter:

1
2
3
4
5
6
7
8
9
10
11
// See `index.js` in [SuperTest] source code.
if ('function' == typeof app) app = http.createServer(app);
// See `lib/test.js` in [SuperTest] source code.
Test.prototype.serverAddress = function (app, path) {
var addr = app.address();
if (!addr) this._server = app.listen(0);
var port = app.address().port;
var protocol = app instanceof htts.Server ? 'https' : 'http';
return protocol + '://1270.0.0.1:' + port + path;
});

Why it does this? This has to do with Express:

1
2
3
4
5
6
7
8
> typeof require('http').createServer()
'object'
> typeof require('express')()
'function'
> typeof require('koa')()
'object'
> require('http').createServer(require('express')()).address()
null

Only when initiating an Express app, it returns a function, not an object. And follows up on lib/test.js, when SuperTest detects the created server is yet to bind to any port number, it will invoke app.listen(0), so called ephemeral port. In fact, it is just a random port.

When something is ephemeral, it last for a very short time. When allowing a server to accept connections, we usually do is setting the server to listen on a specific port:

1
app.listen(3001);

What if setting this to 0 like above or omit this port number?

Locate HTTP Port Number from http.Server Instance

This is a bare minimal HTTP server:

1
require('http').createServer().listen(port);

If the port number is evaluated to be falsy, such as 0, null, undefined or an empty string, a random port will be assigned.

Begin accepting connections on the specified port and hostname. If the hostname is omitted, the server will accept connections on any IPv6 address (::) when IPv6 is available, or any IPv4 address (0.0.0.0) otherwise. A port value of zero will assign a random port. - https://iojs.org/api/http.html#http_server_listen_port_hostname_backlog_callback

We can use command line tool such as netstat to find out the port number:

1
2
$ netstat -ltpn | grep node
tcp6 0 0 :::44055 :::* LISTEN 5624/node

However, it will be very convenient without relying on external tools.

Luckily, we can find out the port number by using the address method:

1
require('http').createServer().listen().address()

The result will be similar to:

1
{ address: '::', family: 'IPv6', port: 44055 }

Furthermore, it works with all popular libraries that create an http.Server instance:

Difference between HTTP HEAD and GET

The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request. - Method Definitions

I was a little bit confused when I encoutered the HTTP status code 204, which states that the response MUST NOT include a message body. The HEAD method contains no body content as well. Should I use 204 instead of 200?

The answer is NO. GET method should always return HTTP status code of 200. Since HEAD method is idential to GET, it should return 200 as well. Just keep in mind that the cost of processing both HEAD and GET requests are almost the same.

HTTP Methods Truth Table

My take on on HTTP methods and resources:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
+-----------------------------------------------------------+
| # | Request-URI | Method | RE | RNE |
+-----------------------------------------------------------+
| 0 | GET /resources | list | 200 | 200 |
| 1 | GET /resources/entity | load/insert | 200 | 404 |
|-----------------------------------------------------------|
| 2 | POST /resources | create | 201 | 409 |
| 3 | POST /resources/entity | N/A | N/A | N/A |
|-----------------------------------------------------------|
| 4 | PUT /resources | (batch) | 200 | 200 |
| 5 | PUT /resources/entity | replace/save | 204 | 201 |
|-----------------------------------------------------------|
| 6 | PATCH /resources | (batch) | 200 | 200 |
| 7 | PATCH /resources/entity | update | 204 | 404 |
|-----------------------------------------------------------|
| 8 | DELETE /resources | (batch) | 200 | 200 |
| 9 | DELETE /resources/entity | remove/delete | 204 | 404 |
+-----------------------------------------------------------+

Notes:

  1. RE: resource exists
  2. RNE: resource not exists
  3. For batch request, whether resource/entity exists or not, the resulting HTTP
    status code is always 200, because the code is used to indicate the status
    of the operation. The actual status code of each entity is enclosed in the
    response array. When there are no matching entities, the response is an empty
    array, therefore, status code 204 is not used.
  4. There are two situation, a new resource is being created, then the Location
    header must indicate the fully qualified resource URI.