jq:Command-line JSON processor

ML
Author

Bhoomi Agrawal, Dhruv Gupta, Antara Donde

Published

February 25, 2025

jq – Lightweight command-line JSON processor

JSON (JavaScript Object Notation) is a data format which structures and stores data efficiently. It is lightweight and readable which means that it is easy to read and write for humans and easy for machines to parse and generate. It is commonly used in APIs for the exchange of data between different applications. It is used in different programming languages like Python, JavaScript, Java, etc. JSON is a library in Python and its object looks like a python dictionary. ##### JSON Data Types: 1. String → “Alice” 2. Number → 25 3. Boolean → true / false 4. Array → [“Math”, “Science”] 5. Object → {“city”: “New York”, “zip”: “10001”} 6. Null → null

Introduction to Jq

Jq is a command-line tool which is used for transforming, processing and the filtering of JSON data. It is fast, readable and has been specially designed for effectively handling structured JSON files. There are different other JSON processors like Jello, jid, Gron and fx which can be used to alter data according to our needs, but jq is considered best because of its speed and flexibility. It can handle structured JSON data directly in the command line, making it a powerful tool for all developers and data analysts. A jq program is a filter. It takes an input and gives an output. It has a lot of different bulletin filters for different tasks. Filters can be glued together in jq, to basically form the loops and iterations that are made in different languages. The input to jq is parsed as a sequence of JSON values which are seperated by white space and are passed through the provied filter one at a time. Then these values are written in the stadard output.

Installation and Setup Guide

Steps to install jq:

The jq could be installed in two major ways. 1. Using system commands/ external package managers 2. Manual Installation from jq official website (https://jqlang.org/download/)

For Windows

  1. Chocolatey (choco):
  • Open PowerShell as Administrator and run:
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
  • Restart PowerShell and check installation:
choco --version
  • Further install jq using:
choco install jq
  1. Winget (winget) (Comes preinstalled on Windows 10/11):
  • If not installed, get it from the Microsoft Store: (https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=IN)
  • Check installation:
winget --version
  • Further install jq using:
 winget install jqlang.jq
  1. Scoop (scoop):
  • Open Powershell as administrator and run:
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
iwr -useb get.scoop.sh | iex
  • Verify Installation:
scoop --version
  • Further install jq using:
scoop install jq 

Alternatively, you can download the pre-built binary for Windows from the jq releases page and add it to your system’s PATH. (https://jqlang.org/download/)

For Linux

The package managers used in this are already pre installed in Linux. Using Bash is appropriate and recommended for Linux users because of its better compatibality, syntax friendliness and it comes pre installed in Linux systems.

  1. APT (apt-get or apt) – Used in Debian-based distributions like Ubuntu, Debian, and Linux Mint.
  • In bash:
sudo apt install jq
  • In some older systems use:
sudo apt-get install apt
  1. DNF (dnf) – Used in Fedora, RHEL (Red Hat Enterprise Linux), and CentOS
  • In bash:
sudo dnf install jq
  • In some older systems use:
sudo yum install jq

Key features and functionalities of jq

Overview- - | jq . → Pretty print JSON - | jq ‘.key’ → Extract a specific key - | jq ‘.[ ].key’ → Extract values from an array - | jq ‘select(.condition)’ → Filter JSON by conditions - | jq ‘keys’ → Get all keys in an object - | jq ‘length’ → Count array elements

This is our input which we are going to use for this block.

import json

with open("users.json", "r") as file:
    users = json.load(file)

print(users)  
{'users': [{'name': 'User1', 'age': 21, 'address': {'city': 'City1', 'zipcode': '12345'}, 'email': 'user1@example.com'}, {'name': 'User2', 'age': 45, 'address': {'city': 'City2', 'zipcode': '67890'}, 'email': 'user2@example.com'}, {'name': 'User3', 'age': 23, 'address': {'city': 'City3', 'zipcode': '54321'}, 'email': 'user3@example.com'}, {'name': 'User4', 'age': 34, 'address': {'city': 'City4', 'zipcode': '98765'}, 'email': 'user4@example.com'}]}

Some of the key features of jq are given below:

  1. Pretty-printing JSON: It formats the JSON data into a form which is in readable form.

    EXAMPLE:

    • command
jq . users.json
  • output
from IPython.core.display import HTML

HTML('<img src="1.png" style="width:100%; display:block; margin:0;">')
  1. Extracting specific fields: It is used to select a speecific field from the JSON object.

Example:

  • command
jq '.users[].age' users.json
  • for nested fields
jq '.users[].address.city' users.json
from IPython.core.display import HTML

HTML('<img src="2new.png" style="width:100%; display:block; margin:0;">')
  1. Filtering of the JSON data

    We can extract the data that matches our condition by iterating through it.

Example:

  • command
jq '.users[] | select(.age > 25)' users.json
  • output
from IPython.core.display import HTML

HTML('<img src="3.png" style="width:100%; display:block; margin:0;">')
  1. Transforming JSON by mapping and modifying

    We can change the structure of a JSON data or can form new values.

Example:

  • command
jq '.users[] | {fullName: .name, years: .age}' users.json
  • output
from IPython.core.display import HTML

HTML('<img src="4new.png" style="width:100%; display:block; margin:0;">')
  1. Working with arrays

    We can use the .[] ,operator to extract values from an array

Example:

  • Command
jq '.users[].email' users.json
from IPython.core.display import HTML

HTML('<img src="5.png" style="width:100%; display:block; margin:0;">')
  1. Sorting JSON Data:

    We can use [jq ‘.JSON_data | sort_by(.(parameter))’ JSON_data.]json to sort the data according to the parameter. This helps us to get the data as per a particular category.

Example:

  • command
jq ' .users | sort_by(.age)' users.json
  • output
from IPython.core.display import HTML

HTML('<img src="6new.png" style="width:100%; display:block; margin:0;">')
  1. Counting elements

    We can print the number of users useing the length command.

Example:

  • command
jq '.users | length' users.json
  • output
from IPython.core.display import HTML

HTML('<img src="7.png" style="width:100%; display:block; margin:0;">')
  1. String manipulation

    This allows us to manipulate of strings using different filters for required output.

Example:

  • command for making names uppercase
jq '.users[].name | ascii_upcase' users.json
  • output
from IPython.core.display import HTML

HTML('<img src="8.png" style="width:100%; display:block; margin:0;">')
  1. Combining multiple commands

    We can form a chain of different commands using the pipe ‘|’ operator. Which means that we could impose different conditions over the data in a single line of code.

Example: Extract names, sort them and format them as an array.

  • command
jq '[.users[].name] | sort' users.json
  • output
from IPython.core.display import HTML

HTML('<img src="9.png" style="width:100%; display:block; margin:0;">')
  1. The select operator This allows us to select data with particular conditions.

Example:

  • command
jq '.users[] | select(.age > 25)' users.json
  • output
from IPython.core.display import HTML

HTML('<img src="10.png" style="width:100%; display:block; margin:0;">')

Most common mistakes!!

These are some mistakes that us being a beginner tend to make and to make sure that you do not commit any of these we take up this topic.

Below we have used the keyword “echo” as it allows us to pass JSON data as input directly rather than providing the whole file for input.

  1. Running jq without JSON input

    Incorrect code-

jq '.name'
Correct code-
echo '{"name": "Alice"}' | jq '.name'
  1. Wrong Structure

    Incorrect code-

echo '{"users": [{"name": "Alice"}]}' | jq '.name'
Correct code-
echo '{"users": [{"name": "Alice"}]}' | jq '.users[0].name'
  1. syntax errors-

    Incorrect code-

echo '{"name": "Alice", "age": 30}' | jq '.age = 35'
Correct code-
echo '{"name": "Alice", "age": 30}' | jq '. | .age = 35'

Use cases

jq is widely used in DevOps, API development, log analysis, data processing and automation. Its speed, flexibility, and simplicity make it one of the best tools for handling JSON data efficiently.

  1. Processing API Responses

    Suppose you are working with APIs that give JSSON data andd you need to extract ceertain fields.

    Steps:

    1. Fetch API data.

    2. Use jq command to extract the required fields.

    EXAMPLE: To extract all ussernames from a JSON API response

  2. Log file analysis

    Suppose many system logs are stored in JSON format, you can use jq to filter specific log details.

    Steps:

    1. Create logs.json

    2. Use jq to filter only errors

     jq '.logs[] | select(.level=="error")' logs.json

    EXAMPLE: To extract error messages from a log file (logs.json)

  3. Data transforming and formatting

    Suppose you need too convert the JSON data into a more structured format like .csv

    Steps:

    1. Use users.json

    2. Convert the JSON data into CSV using

jq -r '.users[] | [.name, .email, .age] | @csv' users.json
EXAMPLE:
To convert the given data into csv.
  1. Combining and merging JSON files

    Suppose you have multiple JSON files and you need to merge them using jq.

    Steps:

     (a) Create two JSON files data1.json and data2.json
    
     (b) Merge them into a single file using 
   jq -s '.[0].users + .[1].users' data1.json data2.json
EXAMPLE:
    Merging data1.json and data2.json
  1. Automating data pipelines

    Supposee using jq in a script to automate JSON data extraction and transformation.

    Steps:

         (a) Extract only emails using
jq -r '.users[].email' users.json > emails.txt
        (b) Check the content of emails.txt using 
cat emails.txt
EXAMPLE:
    To extract user emails and save them to a file.
  1. Configuration file handling

    Suppose you need to moddify JSON based configuration files.

     (a) Create config.json
    {
    "settings": {
    "theme": "dark",
    "notifications": true,
    "enabled": false}
    }
     (b) Change "enabled: false to true using
    jq '.settings.enabled = true' config.json > new_config.json

Conclusion

jq is a powerful yet simple tool for working with JSON in the command line. It helps users quickly read, filter, and modify JSON data with ease. Whether you’re handling API responses, configuration files, or automation tasks, jq makes the process faster and more efficient. Its easy-to-use commands save time and effort, making it a great choice for anyone who works with JSON regularly.

References

  1. jq documentation https://devdocs.io/jq/

  2. https://www.baeldung.com/linux/jq-command-json?utm_source

  3. https://jqlang.org/?utm_source

  4. https://manpages.org/jq/?utm_source

Other manuals and tutorials

  1. https://jqlang.org/tutorial/

  2. https://jqlang.org/manual/

  3. https://jqlang.org/manual/v1.6/