jq:Command-line JSON processor

jq – Lightweight command-line JSON processor

JSON (JavaScript Object Notation) is a data format which structures and stores data efficiently. It is lightweight and readable which means that it is easy to read and write for humans and easy for machines to parse and generate. It is commonly used in APIs for the exchange of data between different applications. It is used in different programming languages like Python, JavaScript, Java, etc. JSON is a library in Python and its object looks like a python dictionary. ##### JSON Data Types: 1. String → “Alice” 2. Number → 25 3. Boolean → true / false 4. Array → [“Math”, “Science”] 5. Object → {“city”: “New York”, “zip”: “10001”} 6. Null → null

Introduction to Jq

Jq is a command-line tool which is used for transforming, processing and the filtering of JSON data. It is fast, readable and has been specially designed for effectively handling structured JSON files. There are different other JSON processors like Jello, jid, Gron and fx which can be used to alter data according to our needs, but jq is considered best because of its speed and flexibility. It can handle structured JSON data directly in the command line, making it a powerful tool for all developers and data analysts. A jq program is a filter. It takes an input and gives an output. It has a lot of different bulletin filters for different tasks. Filters can be glued together in jq, to basically form the loops and iterations that are made in different languages. The input to jq is parsed as a sequence of JSON values which are seperated by white space and are passed through the provied filter one at a time. Then these values are written in the stadard output.

Installation and Setup Guide

Steps to install jq:

The jq could be installed in two major ways. 1. Using system commands/ external package managers 2. Manual Installation from jq official website (https://jqlang.org/download/)

For Windows

1. Chocolatey (choco):

• Open PowerShell as Administrator and run:

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

• Restart PowerShell and check installation:

choco --version

• Further install jq using:

choco install jq

2. Winget (winget) (Comes preinstalled on Windows 10/11):

• If not installed, get it from the Microsoft Store: (https://apps.microsoft.com/detail/9nblggh4nns1?hl=en-US&gl=IN)
• Check installation:

winget --version

• Further install jq using:

 winget install jqlang.jq

3. Scoop (scoop):

• Open Powershell as administrator and run:

Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
iwr -useb get.scoop.sh | iex

• Verify Installation:

scoop --version

• Further install jq using:

scoop install jq 

Alternatively, you can download the pre-built binary for Windows from the jq releases page and add it to your system’s PATH. (https://jqlang.org/download/)

For Linux

The package managers used in this are already pre installed in Linux. Using Bash is appropriate and recommended for Linux users because of its better compatibality, syntax friendliness and it comes pre installed in Linux systems.

1. APT (apt-get or apt) – Used in Debian-based distributions like Ubuntu, Debian, and Linux Mint.

• In bash:

sudo apt install jq

• In some older systems use:

sudo apt-get install apt

1. DNF (dnf) – Used in Fedora, RHEL (Red Hat Enterprise Linux), and CentOS

• In bash:

sudo dnf install jq

• In some older systems use:

sudo yum install jq

Key features and functionalities of jq

Overview- - | jq . → Pretty print JSON - | jq ‘.key’ → Extract a specific key - | jq ‘.[ ].key’ → Extract values from an array - | jq ‘select(.condition)’ → Filter JSON by conditions - | jq ‘keys’ → Get all keys in an object - | jq ‘length’ → Count array elements

This is our input which we are going to use for this block.

import json

with open("users.json", "r") as file:
    users = json.load(file)

print(users)  

{'users': [{'name': 'User1', 'age': 21, 'address': {'city': 'City1', 'zipcode': '12345'}, 'email': 'user1@example.com'}, {'name': 'User2', 'age': 45, 'address': {'city': 'City2', 'zipcode': '67890'}, 'email': 'user2@example.com'}, {'name': 'User3', 'age': 23, 'address': {'city': 'City3', 'zipcode': '54321'}, 'email': 'user3@example.com'}, {'name': 'User4', 'age': 34, 'address': {'city': 'City4', 'zipcode': '98765'}, 'email': 'user4@example.com'}]}

Some of the key features of jq are given below:

1. Pretty-printing JSON: It formats the JSON data into a form which is in readable form.

   EXAMPLE:

   • command

jq . users.json

• output

from IPython.core.display import HTML

HTML('<img src="1.png" style="width:100%; display:block; margin:0;">')

2. Extracting specific fields: It is used to select a speecific field from the JSON object.

Example:

• command

jq '.users[].age' users.json

• for nested fields

jq '.users[].address.city' users.json

from IPython.core.display import HTML

HTML('<img src="2new.png" style="width:100%; display:block; margin:0;">')

3. Filtering of the JSON data

   We can extract the data that matches our condition by iterating through it.

Example:

• command

jq '.users[] | select(.age > 25)' users.json

• output

from IPython.core.display import HTML

HTML('<img src="3.png" style="width:100%; display:block; margin:0;">')

4. Transforming JSON by mapping and modifying

   We can change the structure of a JSON data or can form new values.

Example:

• command

jq '.users[] | {fullName: .name, years: .age}' users.json

• output

from IPython.core.display import HTML

HTML('<img src="4new.png" style="width:100%; display:block; margin:0;">')

5. Working with arrays

   We can use the .[] ,operator to extract values from an array

Example:

• Command

jq '.users[].email' users.json

from IPython.core.display import HTML

HTML('<img src="5.png" style="width:100%; display:block; margin:0;">')

6. Sorting JSON Data:

   We can use [jq ‘.JSON_data | sort_by(.(parameter))’ JSON_data.]json to sort the data according to the parameter. This helps us to get the data as per a particular category.

Example:

• command

jq ' .users | sort_by(.age)' users.json

• output

from IPython.core.display import HTML

HTML('<img src="6new.png" style="width:100%; display:block; margin:0;">')

7. Counting elements

   We can print the number of users useing the length command.

Example:

• command

jq '.users | length' users.json

• output

from IPython.core.display import HTML

HTML('<img src="7.png" style="width:100%; display:block; margin:0;">')

8. String manipulation

   This allows us to manipulate of strings using different filters for required output.

Example:

• command for making names uppercase

jq '.users[].name | ascii_upcase' users.json

• output

from IPython.core.display import HTML

HTML('<img src="8.png" style="width:100%; display:block; margin:0;">')

9. Combining multiple commands

   We can form a chain of different commands using the pipe ‘|’ operator. Which means that we could impose different conditions over the data in a single line of code.

Example: Extract names, sort them and format them as an array.

• command

jq '[.users[].name] | sort' users.json

• output

from IPython.core.display import HTML

HTML('<img src="9.png" style="width:100%; display:block; margin:0;">')

10. The select operator This allows us to select data with particular conditions.

Example:

• command

jq '.users[] | select(.age > 25)' users.json

• output

from IPython.core.display import HTML

HTML('<img src="10.png" style="width:100%; display:block; margin:0;">')

Most common mistakes!!

These are some mistakes that us being a beginner tend to make and to make sure that you do not commit any of these we take up this topic.

Below we have used the keyword “echo” as it allows us to pass JSON data as input directly rather than providing the whole file for input.

1. Running jq without JSON input

   Incorrect code-

jq '.name'

Correct code-

echo '{"name": "Alice"}' | jq '.name'

2. Wrong Structure

   Incorrect code-

echo '{"users": [{"name": "Alice"}]}' | jq '.name'

Correct code-

echo '{"users": [{"name": "Alice"}]}' | jq '.users[0].name'

3. syntax errors-

   Incorrect code-

echo '{"name": "Alice", "age": 30}' | jq '.age = 35'

Correct code-

echo '{"name": "Alice", "age": 30}' | jq '. | .age = 35'

Use cases

jq is widely used in DevOps, API development, log analysis, data processing and automation. Its speed, flexibility, and simplicity make it one of the best tools for handling JSON data efficiently.

1. Processing API Responses

   Suppose you are working with APIs that give JSSON data andd you need to extract ceertain fields.

   Steps:

   1. Fetch API data.

   2. Use jq command to extract the required fields.

   EXAMPLE: To extract all ussernames from a JSON API response

2. Log file analysis

   Suppose many system logs are stored in JSON format, you can use jq to filter specific log details.

   Steps:

   1. Create logs.json

   2. Use jq to filter only errors

    jq '.logs[] | select(.level=="error")' logs.json

   EXAMPLE: To extract error messages from a log file (logs.json)

3. Data transforming and formatting

   Suppose you need too convert the JSON data into a more structured format like .csv

   Steps:

   1. Use users.json

   2. Convert the JSON data into CSV using

jq -r '.users[] | [.name, .email, .age] | @csv' users.json

EXAMPLE:
To convert the given data into csv.

4. Combining and merging JSON files

   Suppose you have multiple JSON files and you need to merge them using jq.

   Steps:

    (a) Create two JSON files data1.json and data2.json
   
    (b) Merge them into a single file using 

   jq -s '.[0].users + .[1].users' data1.json data2.json

EXAMPLE:
    Merging data1.json and data2.json

5. Automating data pipelines

   Supposee using jq in a script to automate JSON data extraction and transformation.

   Steps:

        (a) Extract only emails using

jq -r '.users[].email' users.json > emails.txt

        (b) Check the content of emails.txt using 

cat emails.txt

EXAMPLE:
    To extract user emails and save them to a file.

6. Configuration file handling

   Suppose you need to moddify JSON based configuration files.

    (a) Create config.json

   {
   "settings": {
   "theme": "dark",
   "notifications": true,
   "enabled": false}
   }

    (b) Change "enabled: false to true using

   jq '.settings.enabled = true' config.json > new_config.json

Conclusion

jq is a powerful yet simple tool for working with JSON in the command line. It helps users quickly read, filter, and modify JSON data with ease. Whether you’re handling API responses, configuration files, or automation tasks, jq makes the process faster and more efficient. Its easy-to-use commands save time and effort, making it a great choice for anyone who works with JSON regularly.

References

1. jq documentation https://devdocs.io/jq/

2. https://www.baeldung.com/linux/jq-command-json?utm_source

3. https://jqlang.org/?utm_source

4. https://manpages.org/jq/?utm_source

Other manuals and tutorials

1. https://jqlang.org/tutorial/

2. https://jqlang.org/manual/

3. https://jqlang.org/manual/v1.6/