Running into an error when executing an AWS command:

1
2
3
4

$ aws ec2 describe-instances
An error occurred (AuthFailure) when calling the DescribeInstances operation: AWS
was not able to validate the provided access credentials

From the error message, it appears to be an error with access credentials. But after updating to a new credential, and even updated the AWS package, the error still persisted. After trying out other commands, there was an error message containing “signature not yet current” with timestamps. So, the actual problem was due to inaccurate local clock. Hence, the solution is to sync the local date and time by polling the Network Time Protocol (NTP) server:

1

$ sudo ntpdate pool.ntp.org

ntpdate can be run manually as necessary to set the host clock, or it can be run from the host startup script to set the clock at boot time. This is useful in some cases to set the clock initially before starting the NTP daemon ntpd. It is also possible to run ntpdate from a cron script. However, it is important to note that ntpdate with contrived cron scripts is no substitute for the NTP daemon, which uses sophisticated algorithms to maximize accuracy and reliability while minimizing resource use. Finally, since ntpdate does not discipline the host clock frequency as does ntpd, the accuracy using ntpdate is limited.[^1]

From the description, we can learn that we can make things even easier by installing NTP package:

1

$ sudo apt-get install -y ntp

Network Time Protocol daemon and utility programs NTP, the Network Time Protocol, is used to keep computer clocks accurate by synchronizing them over the Internet or a local network, or by following an accurate hardware receiver that interprets GPS, DCF-77, NIST or similar time signals.[^2]

Verify the installation and execution:

1
2

$ ps -e | grep ntpd
 4964 ?        00:00:00 ntpd

with the environment:

1
2

$ aws --version
aws-cli/1.10.53 Python/2.7.6 Linux/3.13.0-92-generic botocore/1.4.43

[^1]: $ man nptdate
[^2]: $ apt-cache show ntp

To determine whether your Vagrant is out of date or not, issue:

1
2
3
4
5

$ vagrant version
Installed Version: 1.7.4
Latest Version: 1.7.4
You're running an up-to-date version of Vagrant!

If yours is out of date, you can download it from its download page. However, being a lazy programmer, that’s still too many clicks. I want to figure out a way to update or install Vagrant from the command line without prompt.

And here you go!

Vagrant binaries are distributed by Bintray, its format is:

1

https://dl.bintray.com/mitchellh/vagrant/vagrant_{version}.msi

For example:

1

https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.4.msi

After you issue vagrant version, you know the correct version to upgrade, then download it:

1

$ curl -sSL https://dl.bintray.com/mitchellh/vagrant/vagrant_1.7.4.msi > vagrant.msi

Install it by msiexec:

1

$ msiexec /i C:\Users\Chao\Downloads\vagrant.msi /passive /norestart

Make sure to use the full path instead of relative .\vagrant.msi, and Vagrant should be installed in the default location C:\HashiCorp\Vagrant. To find out all possible options, Just type msiexec command only.

Saving a few clicks!

io.js v1.6.0 introduces a new command line option: -r or --require, which can be used to pre-load modules at startup, see:

https://github.com/iojs/io.js/blob/v1.x/CHANGELOG.md#2015-03-19-version-160-chrisdickinson

I can think of one interesting use case: overriding the default behavior or console.log to perform deep inspection:

1
2
3
4
5
6
7

// console.js
var util = require('util');
var log = console.log;
console.log = function (obj) {
  log(util.inspect(obj, { colors: true, depth: null }));
};

Now I can pre-load this script to have all console log show in better format:

1

$ iojs -r /path/to/console.js app.js

Time to find out more use cases.

Need to know your public IP address? Use one of the following:

1
2
3
4
5
6

$ curl -s checkip.amazonaws.com
$ curl -s ipinfo.io/ip
$ curl -s www.trackip.net/ip
$ curl -s whatsmyip.me
$ curl -s whatismyip.akamai.com
$ curl -s icanhazip.com

I always forget about how to use shell environment variable correctly. I think because I did not grasp the key concept. For example, here is the wrong approach:

1
2

$ FOO=foo BAR=bar echo foo is ${FOO} and bar is ${BAR}
foo is and bar is

The correct answer I was hoping for is:

1

foo is foo and bar is bar

Because what I frequently do is use environment variable inside shell script:

1
2
3
4
5

#!/usr/bin/env bash
#
# script.sh
echo foo is ${FOO} and bar is ${BAR}

then do:

1
2

$ FOO=foo BAR=bar ./script.sh
foo is foo and bar is bar

This works as expected because the environment variables are passed into the subshell that executing the script.

But why not the first or the wrong approach?

The reason is that shell expands the variable before the command being executed. To shell, the first approach looks like:

1

$ FOO=foo BAR=bar echo foo is and bar is

The environment variables FOO and BAR were expanded before the command was executed. Hence, no values were printed.

In Amazon S3 CLI, there are only a handful commands:

1

cp      ls      mb      mv      rb      rm      sync    website

We can use cp command to retrieve S3 object:

1

$ aws s3 cp s3://mybucket/myfile.json myfile.json

But if only the metadata of the object, such as ETag or Content-Type is needed, the S3 CLI does not have any command to do that.

Now enter S3API CLI. Not only the CLI commands can retrieve S3 objects, but also associated metadata. For example, retrieve S3 object similar to aws s3 cp:

1

$ aws s3api get-object --bucket mybucket --key myfile.json

Just need the metadata of the object, use head-object, which retrieves metadata without the object itself, as HTTP HEAD method:

If you work from place to place, such as from one coffee shop to another, and you need access to your Amazon EC2 instances, but you don’t want to allow traffics from all IP addresses. You can use the EC2 Security Groups to allow the IP addresses from those locations. But once you move on to a different location, you want to delete the IP address from the previous location. The process to do these manually and over and over again quickly becomes cumbersome. Here is a command line method that quickly removes all other locations and allows only the traffic from your current location.

The steps are:

1. Revoke all existing sources to a particular port
2. Grant access to the port only from the current IP address

Assume the following:

• Profile: default
• Security group: mygroup
• Protocol: tcp
• Port: 22

First, revoke access to the port from all IP addresses:

1
2
3
4
5
6
7
8
9
10
11

$ aws ec2 describe-security-groups \
    --profile default \
    --group-names mygroup \
    --query 'SecurityGroups[0].IpPermissions[?ToPort==`22`].IpRanges[].CidrIp' | \
  jq .[] | \
  xargs -n 1 aws ec2 revoke-security-group-ingress \
    --profile default \
    --group-name mygroup \
    --protocol tcp \
    --port 22 \
    --cidr

The aws ec2 describe-security-groups command before the first pipe returns JSON formatted data, filtered via JMESPath query, which is supported by AWS CLI, for example:

1
2
3
4

[
  "XXX.XXX.XXX.XXX/32",
  "XXX.XXX.XXX.XXX/32"
]

jq command simply converts an array of JSON to line by line strings, which xarg takes in, loops through and deletes one IP address at a time.

After this step, all IP addresses originally allowed are all revoked. Next step is to grant access to the port from a single IP address:

Amazon SQS or Simple Queue Service is a fast, reliable, scalable, fully managed message queuing service. There is also AWS CLI or Command Line Interface available to use with the service.

If you have a lot of messages in a queue, this command will show the approximate number:

1
2
3

$ aws sqs get-queue-attributes \
  --queue-url $url \
  --attribute-names ApproximateNumberOfMessages

Where $url is the URL to the Amazon SQS queue.

There is no command to delete all messages yet, but you can chain a few commands together to make it work:

What is the best command line tool to process JSON?

Hmm… Okay, let’s try different command line JSON processing tools with the following use case to decide which one is the best to use.

Here is the use case: JSON | filter | shell. A program outputs JSON data, pipes into a JSON command line processing tool to filter data, and then send to a shell command to do more work.

Here is a snippet of sample JSON data:

1
2
3
4
5
6
7
8
9
10
11
12

[
  {
    "id": 1,
    "username": "foo",
    "name": "Foo Foo"
  },
  {
    "id": 2,
    "username": "bar",
    "name": "Bar Bar"
  }
]

Or in one-liner data.json:

1

[{"id":1,"username":"foo","name":"Foo Foo"},{"id":2,"username":"bar","name":"Bar Bar"}]

The command line JSON processor should filter each element of the array and convert it into its own line:

1
2

{"name":"Foo Foo"}
{"name":"Bar Bar"}

The result will be piped as the input line by line into a shell script echo.bash:

1
2
3
4
5

#!/usr/bin/env bash
while read line; do
  echo "ECHO: '"$line"'"
done

The final output should be:

1
2

ECHO: '{"name":"Foo Foo"}'
ECHO: '{"name":"Bar Bar"}'

Custom Solution

Before start looking for existing tools, let’s see how difficult it is to write a custom solution.

1
2
3
4
5
6
7
8
9
10
11
12
13

// Filter and convert array element into its own line.
var rl = require('readline').createInterface({
  input : process.stdin,
  output: process.stdout,
});
rl.on('line', function (line) {
  JSON.parse(line).forEach(function (item) {
    console.log('{"name":"' + item.name + '"}');
  });
}).on('close', function () {
  // Shh...
});

Perform a test run:

1
2
3

$ cat data.json | node filter.js  | bash echo.bash 
ECHO: '{"name":"Foo Foo"}'
ECHO: '{"name":"Bar Bar"}'

Well, it works. In essence, we are writing a simple JSON parser. Unless you want to keep the footprint small, you don’t want to write another JSON parser. And why bother to reinvent the wheel? Let’s start look at the existing solutions.

Node Modules

Let’s start with the tools from NPM registry:

$ npm search json command

Here are a few candidates that appears to be matching from the description:

• jku - Jku is a command-line tool to filter and/or modifiy a JSON stream. It is heavily inspired by jq. (2 stars and not active, last update 8 months ago).
• json or json-command - JSON command line procesing toolkit. (122 stars and 14 forks, last update 9 months ago)
• jutil - Command-line utilities for manipulating JSON. (88 stars and 2 forks, last update more than 2 years ago)

Not a lot of choice, and modules are not active. This might be that because there is already a really good solution, jq, which has 2493 stars and 145 forks, and the last update was 6 days ago.

jq

jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text. - jq

Instead of NPM install, do:

$ sudo apt-get -y install jq

Since we don’t need color or prettified, just line by line. So, here is the command chain:

1
2
3

$ cat data.json | jq -c -M '.[] | {name}'  | bash echo.bash
ECHO: '{"name":"Foo Foo"}'
ECHO: '{"name":"Bar Bar"}'

jq can do much more than just the example just shown. It has zero runtime dependencies, and flexible to deal with not just array but object as well.

Conclusion

jq is clearly the winner here, with the least dependency, the most functionality and more popularity, as well as a comprehensive documentation.

This is a brief guide for installing AWS Command Line Interface (CLI) on Ubuntu Linux.

The AWS Command Line Interface (CLI)] is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts. [2]

The point here is unified, one tool to run all Amazon AWS services.

Install

The installation procedure applies to Ubuntu Linux with Zsh and Bash.

Install pip, a Python package manager:

$ sudo apt-get install python-pip

Install awscli:

$ sudo pip install awscli

Install autocompletion:

$ which aws_zsh_completer.sh
/usr/local/bin/aws_zsh_completer.sh
$ source aws_zsh_completer.sh

Add this line to ~/.zshrc as well.

Or for Bash ~/.bashrc:

$ echo '\n# Enable AWS CLI autocompletion' >> ~/.bashrc
$ echo 'complete -C aws_completer aws' >> ~/.bashrc
$ source ~/.bashrc

Test installation:

$ which aws
/usr/local/bin/aws
$ aws help

Test autocompletion:

$ aws <tab><tab>

You should see a list of all available AWS commands.

Usage

Before using aws-cli, you need to tell it about your AWS credentials. There are three ways to specify AWS credentials:

1. Environment variables
2. Config file
3. IAM Role

Using config file is preferred, which is a simple ini file format to be stored in ~/.aws/config. A soft link can be used to link it or just tell awscli where to find it:

$ export AWS_CONFIG_FILE=/path/to/config_file

It is better to use IAM roles with any of the AWS services:

The final option for credentials is highly recommended if you are using aws-cli on an EC2 instance. IAM Roles are a great way to have credentials installed automatically on your instance. If you are using IAM Roles, aws-cli will find them and use them automatically. [4]

The default output is in JSON format. Other formats are tab-delimited text and ASCII-formatted table. For example, using --query filter and table output:

$ aws ec2 describe-instances --query 'Reservations[*].Instances[*].{ \
  ID:InstanceId, TYPE:InstanceType, ZONE:Placement.AvailabilityZone, \
  SECURITY:SecurityGroups[0].GroupId, KEY:KeyName, VPC:VpcId, \
  STATE:State.Name}' --output table

This will print a nice looking table of all EC2 instances.

The command line options also accept JSON format. But when passing in large blocks of data, referring a JSON file is much easier. Both local file and remote URL can be used.

Upgrade

Check the installed and the latest versions:

$ pip search awscli

Upgrade AWS CLI to the latest version:

$ sudo pip install --upgrade awscli

References

1. AWS Command Line Interface
2. User Guide
3. Reference
4. GitHub repository