Linux

Setting Up Local DNS Server for Multi-Devices

As always, the first thing you have to ask yourself is that why do you want to set up a DNS server? Here are a few of mine:

  • Access web services via custom domains instead of IP addresses
  • Update multiple devices (including mobile devices) is time consuming and inconvenient
  • Router does not provide a DNS server

I run many web services in the local network, for example, GitLab for code repository and Ghost for blogging. Instead of typing IP address for each service, it will be much easier to use custom domain names, for example, accessing my blog via http://ghost/, not dot com nor localhost.

This can be easily done if there is just one machine. You can update the host configuration file in the local machine such as /etc/hosts in Linux or C:\Windows\System32\drivers\etc\hosts in Windows. But I have multiple devices: a laptop, a tablet, and of course a smartphone. Updating multiple devices is a pain. And for many mobile devices running systems such as iOS or Android, it is not easy to edit the host configuration file without rooting the devices. Therefore, we will opt to update a single DNS configuration file.

DNS configuration can be inherited from the DHCP server from the local network router. Unfortunately, the router I have does not provide a built-in DNS server. I have to setup my own DNS server. Once the DNS server has been setup, the router will use the server as the primary DNS server, and falls back to default gateway IP as the secondary DNS server, or we can use either Google’s public DNS server with IP address 8.8.8.8 or any other alternatives.

  • Primary DNS server: 192.168.0.100
  • Secondary DNS server: 192.168.0.1, 8.8.8.8 or others

I am going to setup a DNS server in a Debian Jessie machine. Here is a summary of steps:

  1. Install BIND
  2. Add custom zone

Get Started with the Cut Command of GNU Coreutils

A part of GNU Coreutils, cut command likes paste and join is one that operates on fields. The cut command prints selected parts (fields or sections) of lines.

Things to keep in mind:

  • Work on text stream line by line
  • Print sections/parts/fields
  • Either -c, -b or -f option must be used
  • The default delimiter is TAB
  • The delimiter must be a single character
  • Consecutive delimiters need to be consolidated into one
  • Cut multiple or a range of fields

Generate sample data and saved in the file tab.txt:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ file=tab.txt && \
for i in $(seq 1 9); do \
for j in $(seq 1 9); do \
if [ "$j" == "9" ]; then \
echo -e "$i$j"; \
else \
echo -en "$i$j\t"; \
fi; \
done; \
done > $file && cat $file
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

Fields on each line is separated by a tab character.

There is a required option:

1
2
3
$ cut tab.txt
cut: you must specify a list of bytes, characters, or fields
Try 'cut --help' for more information.

These options are:

  • bytes: -b or --bytes=LIST
  • characters: -c or --characters=LIST
  • fields: -f or --fields=LIST

This is very odd. Option means optional, a default should be provided. But let’s focus on the common used one: fields.

Cut the first and the ninth fields with the default delimiter TAB:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ cut -f 1 tab.txt
11
21
31
41
51
61
71
81
91
$ cut -f 9 tab.txt
19
29
39
49
59
69
79
89
99

Use space as the delimiter:

1
2
3
4
5
6
7
8
9
10
$ cp tab.txt space.txt && sed -i 's/\t/ /g' $_ && cat $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

We must choose a different delimiter via -d or --delimiter=DELIM option:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ cut -f 1 $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99
$ cut -f 1 -d ' ' $_
11
21
31
41
51
61
71
81
91

Delimiter must be a single character:

1
2
3
$ cut -f 1 -d '\s' $_
cut: the delimiter must be a single character
Try 'cut --help' for more information.

Files containing mixed delimiters (tab and space):

1
2
3
4
5
6
7
8
9
10
$ cp tab.txt mixed.txt && sed -i 's/\(9.\)\t/\1 /g' $_ && cat $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

will print any line that contains no delimiter character:

1
2
3
4
5
6
7
8
9
10
$ cut -f 1 mixed.txt
11
21
31
41
51
61
71
81
91 92 93 94 95 96 97 98 99

Or use -s or --only-delimited option to omit those lines.

1
2
3
4
5
6
7
8
9
$ cut -sf 1 mixed.txt
11
21
31
41
51
61
71
81

But the better approach is to do data cleansing prior.

What about multiple TAB characters in the file:

1
2
3
4
5
6
7
8
9
10
$ sed -i 's/\(1.\)\t/\1\t\t/' mixed.txt && cat $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

An empty field is still a field:

1
2
3
4
5
6
7
8
9
$ cut -sf 2 mixed.txt
22
32
42
52
62
72
82

Therefore, the drawback here is that there cannot be multiple delimiter sticking together. Must perform data cleansing to reduce consecutive delimiters into a single one:

1
2
3
4
5
6
7
8
9
10
$ sed -i 's/\t\+/\t/g' mixed.txt && cat $_
11 12 13 14 15 16 17 18 19
21 22 23 24 25 26 27 28 29
31 32 33 34 35 36 37 38 39
41 42 43 44 45 46 47 48 49
51 52 53 54 55 56 57 58 59
61 62 63 64 65 66 67 68 69
71 72 73 74 75 76 77 78 79
81 82 83 84 85 86 87 88 89
91 92 93 94 95 96 97 98 99

Multiple fields can be cut:

1
2
3
4
5
6
7
8
9
10
$ cut -f 1,3,5,7,9 tab.txt
11 13 15 17 19
21 23 25 27 29
31 33 35 37 39
41 43 45 47 49
51 53 55 57 59
61 63 65 67 69
71 73 75 77 79
81 83 85 87 89
91 93 95 97 99

Cut a range:

1
2
3
4
5
6
7
8
9
10
$ cut -f 3-5 tab.txt
13 14 15
23 24 25
33 34 35
43 44 45
53 54 55
63 64 65
73 74 75
83 84 85
93 94 95

Cut up to or from a field:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ cut -f -3 tab.txt
11 12 13
21 22 23
31 32 33
41 42 43
51 52 53
61 62 63
71 72 73
81 82 83
91 92 93
$ cut -f 7- tab.txt
17 18 19
27 28 29
37 38 39
47 48 49
57 58 59
67 68 69
77 78 79
87 88 89
97 98 99

When cut multiple fields, the fields are separated by the same delimiter used (indicated by -d field or TAB as the default). If change the output delimiter, it’s not the job of cut, pipe to another program:

1
2
3
4
5
6
7
8
9
10
$ cut -f 3-5 tab.txt | sed 's/\t/ /g'
13 14 15
23 24 25
33 34 35
43 44 45
53 54 55
63 64 65
73 74 75
83 84 85
93 94 95

cURL: Read from File or Stdin

Redirect or pipe the content of a file or the output of another command as the input of cURL, for example, making a POST request with cURL.

This is done by prefixing the data parameter with @, the rest is the filename to read data from. With -, the data is coming from stdin:

1
2
3
4
5
6
7
8
$ seq 17000 1 17001 | \
curl \
--request POST \
--include \
--header 'Content-Type: application/json' \
--data-binary @- \
--no-buffer \
https://example.com/data

Command seq 17000 1 17001 returns:

1
2
17000
17001

The size of the above two numbers is 12 bytes (including two newlines). But when reading data via @ with --data option, “carriage returns and newlines will be stripped out” [1]. If you check your server, you will see the content length is only 10 bytes.

To correct this problem, we will send the raw data using --data-binary option instead. This preserves \n and \r characters.

How to Correctly Use Environment Variables on the Command Line

I always forget about how to use shell environment variable correctly. I think because I did not grasp the key concept. For example, here is the wrong approach:

1
2
$ FOO=foo BAR=bar echo foo is ${FOO} and bar is ${BAR}
foo is and bar is

The correct answer I was hoping for is:

1
foo is foo and bar is bar

Because what I frequently do is use environment variable inside shell script:

1
2
3
4
5
#!/usr/bin/env bash
#
# script.sh
echo foo is ${FOO} and bar is ${BAR}

then do:

1
2
$ FOO=foo BAR=bar ./script.sh
foo is foo and bar is bar

This works as expected because the environment variables are passed into the subshell that executing the script.

But why not the first or the wrong approach?

The reason is that shell expands the variable before the command being executed. To shell, the first approach looks like:

1
$ FOO=foo BAR=bar echo foo is and bar is

The environment variables FOO and BAR were expanded before the command was executed. Hence, no values were printed.