Using jq with AWS

Introduction

jq (https://stedolan.github.io/jq/) is an open-source tool designed to parse, filter, display, and transform the contents of JSON input. While working with awscli, we often need to manipulate the json output, filtering for fields, or searching for values. This post is a collection of jq commands that I use repeatedly with awscli.

jq Commands

General

• Extracting a single filed from lists. For example: List instance Ids

aws ec2 describe-instances | jq -r '.Reservations[].Instances[].InstanceId'

• Extracting multiple fields:

$ aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | .InstanceId, .ImageId, .InstanceType, .PrivateIpAddress, .State.Name'  

Output:

i-0fae91112f2404699
ami-00f794a89cf3d1543
t2.micro
10.70.2.28
running
i-0576bd80dda9d1b78
ami-080892cd3615a77ef
t3.xlarge
10.200.0.25
stopped

• Select Resources based on Name Tag:

  aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" "Name=tag:Name,Values=<VALUES-WILDCARD-ALLOWED>" \
  | jq -r ".Reservations[].Instances[].InstanceId"

• Wildcard can be used in the Values field above

Manipulating Output

• List:

aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | [.InstanceId, .ImageId, .InstanceType, .PrivateIpAddress, .State.Name] '

Output:
[
  "i-0fae91112f2404699",
  "ami-00f794f8acf3d8799",
  "t2.micro",
  "10.40.2.238",
  "running"
]
[
  "i-0576bd80dda9d1b78",
  "ami-080efe31d615a5895",
  "t3.xlarge",
  "10.54.0.247",
  "stopped"
]

• CSV format:

aws ec2 describe-instances \
| jq -r '.Reservations[].Instances[] | [.InstanceId, .ImageId, .InstanceType, .PrivateIpAddress, .State.Name] | @csv'  

Output:
"i-0fae91112f2404699","ami-00f794f8acf3d8799","t2.micro","10.40.2.238","running"
"i-0576bd80dda9d1b78","ami-080efe31d615a5895","t3.xlarge","10.54.0.247","stopped"

• Table:

aws ec2 describe-instances | \
jq -r '.Reservations[].Instances[] | [.InstanceId, .InstanceType, .PrivateIpAddress, .State.Name] | @tsv' 

Output:
i-0fae91112f2404699     t2.micro        10.40.2.238     running
i-0576bd80dda9d1b78     t3.xlarge       10.54.0.247     stopped

Cloudformation

Resources:

It is often a good idea to output the names of resources created by a Cloudformation template. If such outputs are available, they can be fitched with the following command (No jq is needed here):

  aws cloudformation describe-stacks --stack-name <STACK_NAME> \
  --query "Stacks[].Outputs[?OutputKey==\`<OUTPUT_NAME>\`].OutputValue" \
  --output text

If that is not the case, then we still can fetch these using describe-stack-resources.

1. If you know the exact name of the resource:

aws cloudformation describe-stack-resources --stack-name <STACK_NAME> \
| jq -r '.StackResources[] | select(.PhysicalResourceId=="<EXACT_RESOURCE_NAME>") | .PhysicalResourceId'

1. If you only have part of the name (maybe the rest was randomized):

 aws cloudformation describe-stack-resources --stack-name <STACK_NAME> \
 | jq -r '.StackResources[].PhysicalResourceId' \
 | grep <PART_OF_RESOURCE_NAME>