JSON (JavaScript Object Notation) is a data format which structures and stores data efficiently. It is lightweight and readable which means that it is easy to read and write for humans and easy for machines to parse and generate. It is commonly used in APIs for the exchange of data between different applications. It is used in different programming languages like Python, JavaScript, Java, etc. JSON is a library in Python and its object looks like a python dictionary. ##### JSON Data Types: 1. String → “Alice” 2. Number → 25 3. Boolean → true / false 4. Array → [“Math”, “Science”] 5. Object → {“city”: “New York”, “zip”: “10001”} 6. Null → null
Introduction to Jq
Jq is a command-line tool which is used for transforming, processing and the filtering of JSON data. It is fast, readable and has been specially designed for effectively handling structured JSON files. There are different other JSON processors like Jello, jid, Gron and fx which can be used to alter data according to our needs, but jq is considered best because of its speed and flexibility. It can handle structured JSON data directly in the command line, making it a powerful tool for all developers and data analysts. A jq program is a filter. It takes an input and gives an output. It has a lot of different bulletin filters for different tasks. Filters can be glued together in jq, to basically form the loops and iterations that are made in different languages. The input to jq is parsed as a sequence of JSON values which are seperated by white space and are passed through the provied filter one at a time. Then these values are written in the stadard output.
Installation and Setup Guide
Steps to install jq:
The jq could be installed in two major ways. 1. Using system commands/ external package managers 2. Manual Installation from jq official website (https://jqlang.org/download/)
Alternatively, you can download the pre-built binary for Windows from the jq releases page and add it to your system’s PATH. (https://jqlang.org/download/)
For Linux
The package managers used in this are already pre installed in Linux. Using Bash is appropriate and recommended for Linux users because of its better compatibality, syntax friendliness and it comes pre installed in Linux systems.
APT (apt-get or apt) – Used in Debian-based distributions like Ubuntu, Debian, and Linux Mint.
In bash:
sudo apt install jq
In some older systems use:
sudo apt-get install apt
DNF (dnf) – Used in Fedora, RHEL (Red Hat Enterprise Linux), and CentOS
In bash:
sudo dnf install jq
In some older systems use:
sudo yum install jq
Key features and functionalities of jq
Overview- - | jq . → Pretty print JSON - | jq ‘.key’ → Extract a specific key - | jq ‘.[ ].key’ → Extract values from an array - | jq ‘select(.condition)’ → Filter JSON by conditions - | jq ‘keys’ → Get all keys in an object - | jq ‘length’ → Count array elements
This is our input which we are going to use for this block.
from IPython.core.display import HTMLHTML('<img src="4new.png" style="width:100%; display:block; margin:0;">')
Working with arrays
We can use the .[] ,operator to extract values from an array
Example:
Command
jq '.users[].email' users.json
from IPython.core.display import HTMLHTML('<img src="5.png" style="width:100%; display:block; margin:0;">')
Sorting JSON Data:
We can use [jq ‘.JSON_data | sort_by(.(parameter))’ JSON_data.]json to sort the data according to the parameter. This helps us to get the data as per a particular category.
Example:
command
jq ' .users | sort_by(.age)' users.json
output
from IPython.core.display import HTMLHTML('<img src="6new.png" style="width:100%; display:block; margin:0;">')
Counting elements
We can print the number of users useing the length command.
Example:
command
jq '.users | length' users.json
output
from IPython.core.display import HTMLHTML('<img src="7.png" style="width:100%; display:block; margin:0;">')
String manipulation
This allows us to manipulate of strings using different filters for required output.
Example:
command for making names uppercase
jq '.users[].name | ascii_upcase' users.json
output
from IPython.core.display import HTMLHTML('<img src="8.png" style="width:100%; display:block; margin:0;">')
Combining multiple commands
We can form a chain of different commands using the pipe ‘|’ operator. Which means that we could impose different conditions over the data in a single line of code.
Example: Extract names, sort them and format them as an array.
command
jq '[.users[].name] | sort' users.json
output
from IPython.core.display import HTMLHTML('<img src="9.png" style="width:100%; display:block; margin:0;">')
The select operator This allows us to select data with particular conditions.
Example:
command
jq '.users[] | select(.age > 25)' users.json
output
from IPython.core.display import HTMLHTML('<img src="10.png" style="width:100%; display:block; margin:0;">')
Most common mistakes!!
These are some mistakes that us being a beginner tend to make and to make sure that you do not commit any of these we take up this topic.
Below we have used the keyword “echo” as it allows us to pass JSON data as input directly rather than providing the whole file for input.
jq is widely used in DevOps, API development, log analysis, data processing and automation. Its speed, flexibility, and simplicity make it one of the best tools for handling JSON data efficiently.
Processing API Responses
Suppose you are working with APIs that give JSSON data andd you need to extract ceertain fields.
Steps:
Fetch API data.
Use jq command to extract the required fields.
EXAMPLE: To extract all ussernames from a JSON API response
Log file analysis
Suppose many system logs are stored in JSON format, you can use jq to filter specific log details.
Steps:
Create logs.json
Use jq to filter only errors
jq '.logs[] | select(.level=="error")' logs.json
EXAMPLE: To extract error messages from a log file (logs.json)
Data transforming and formatting
Suppose you need too convert the JSON data into a more structured format like .csv
jq is a powerful yet simple tool for working with JSON in the command line. It helps users quickly read, filter, and modify JSON data with ease. Whether you’re handling API responses, configuration files, or automation tasks, jq makes the process faster and more efficient. Its easy-to-use commands save time and effort, making it a great choice for anyone who works with JSON regularly.