BudiBadu Logo
Samplebadu

Shell Script by Example: Pipelines

Bash 5.x

Connecting commands together with this sample code showing the pipe operator for chaining utilities, combining grep, sort, and head for data processing, using xargs to convert stdin to arguments, and set -o pipefail for robust error handling.

Code

#!/bin/bash

# Basic pipe: ls output -> grep input
ls -l | grep ".txt"

# Chaining multiple commands
# 1. List processes
# 2. Sort by CPU usage
# 3. Take top 5
ps aux | sort -rnk 3 | head -n 5

# Counting lines
cat file.txt | wc -l

# Using xargs to pass output as arguments
find . -name "*.log" | xargs rm

# Pipefail (error handling)
set -o pipefail
ls /nonexistent | grep "foo"
echo "Exit status: $?" # Will be non-zero due to ls failing

Explanation

Pipelines using the | operator are fundamental to Unix philosophy, connecting the standard output of one command to the standard input of another. This enables building complex data processing workflows by chaining together small, single-purpose utilities. Each command in the pipeline runs concurrently, with data flowing through the pipe as it's produced, making pipelines efficient for processing large datasets.

Common pipeline commands include grep for pattern matching, sort for ordering lines, head and tail for selecting first or last lines, cut for extracting columns, awk for text processing, and sed for stream editing. These utilities form a powerful toolkit for data manipulation directly in the terminal. The xargs command is particularly useful for converting standard input into command-line arguments, enabling commands that don't read from stdin to work in pipelines.

By default, a pipeline's exit status is the exit status of the last command only. This can hide errors if an earlier command fails while the final command succeeds. Enabling set -o pipefail changes this behavior so the pipeline returns a failure status if any command in the chain fails. This is crucial for robust scripting, especially in automated environments where silent failures could cause data corruption or missed errors. Combine this with set -e to make scripts exit immediately on any error.

Code Breakdown

4
| connects ls stdout to grep stdin for filtering output.
10
Three-command pipeline sorts processes by CPU and shows top 5.
16
xargs rm converts filenames from stdin into arguments for rm.
19
set -o pipefail makes pipeline fail if any command fails, not just last.