Pipes allow you to connect multiple commands, sending the output of one command as input to the next.
Pipe Operator (|)
Syntax
Description
The pipe operator (|) takes the standard output of the command on the left and passes it as standard input to the command on the right.
Command 1 executes
The first command runs and produces output.
Output is captured
Instead of displaying on screen, output is captured in a pipe buffer.
Command 2 receives input
The second command reads from the pipe as if reading from a file.
Results display
The final command’s output is shown on screen.
Basic Pipe Examples
Simple Two-Command Pipes
# List files and search for .txt files
ls -la | grep .txt
# Show processes and search for specific process
ps aux | grep minishell
# Count files in directory
ls | wc -l
# Display file and page through content
cat large_file.txt | less
Filtering Output
# Find error messages in logs
cat logfile.txt | grep error
# Search environment for specific variable
env | grep PATH
# Filter and count
ls -la | grep ".c$" | wc -l
Chaining Multiple Commands
You can chain more than two commands together:
# Three commands
ls -la | grep .c | sort
# Four commands
cat file.txt | grep "error" | sort | uniq
# Five commands
ps aux | grep python | awk '{print $2}' | sort -n | head -5
Practical Multi-Pipe Examples
# Find, filter, sort, and count unique items
cat data.txt | grep "user" | sort | uniq | wc -l
# Process log files
cat access.log | grep "404" | awk '{print $1}' | sort | uniq -c | sort -nr
# System monitoring
ps aux | awk '{print $3}' | sort -rn | head -10
Minishell supports unlimited pipe chaining. Each command in the pipeline runs in its own process, allowing true parallel execution.
How Pipes Work
Data Flow
command1 | command2 | command3
[command1] --stdout--> [pipe1] --stdin--> [command2] --stdout--> [pipe2] --stdin--> [command3] --stdout--> [terminal]
Process Creation
When you execute a pipeline:
Pipeline parsing
Minishell counts the pipes to determine how many commands to execute.
Process creation
Each command gets its own child process (via fork()).
Pipe creation
Pipes are created to connect each pair of commands.
Execution
All commands run simultaneously, passing data through pipes.
Waiting
Minishell waits for all child processes to complete.
The number of child processes equals the number of commands (pipes + 1). For example, cmd1 | cmd2 | cmd3 creates 3 processes.
Combining Pipes with Redirections
Pipes and redirections can be used together for powerful combinations:
# Read from file, pipe to command
< input.txt grep "error" | sort
# Equivalent to:
cat input.txt | grep "error" | sort
# Multiple filters
< data.txt grep "user" | grep "active" | wc -l
Output Redirection with Pipes
# Pipe commands, save final output
ls -la | grep .txt > text_files.txt
# Multi-stage processing
cat data.txt | grep "error" | sort > sorted_errors.txt
# Append to file
ps aux | grep python >> process_log.txt
# Read from file, process, write to file
< input.txt sort | uniq > output.txt
# Complex processing
< logs.txt grep "ERROR" | sort | uniq -c > error_summary.txt
Heredoc with Pipes
# Pipe heredoc through commands
cat << EOF | grep "error" | wc -l
info: starting
error: connection failed
warning: retrying
error: timeout
info: stopping
EOF
# Output: 2
# Multiple pipes
cat << EOF | sort | uniq
apple
banana
apple
cherry
banana
EOF
# Output:
# apple
# banana
# cherry
Redirections in the Middle
# Save intermediate results
cat file.txt | grep "error" > errors.txt | wc -l
# Redirect specific commands
ls -la > all_files.txt | grep ".c" > c_files.txt
Each command in a pipeline can have its own redirections. The redirection applies only to that specific command, not the entire pipeline.
Built-in Commands with Pipes
Built-in commands work in pipes:
# Export and filter
export | grep PATH
# Environment variables
env | grep USER | sort
# Echo through pipe
echo "Hello World" | grep World
# PWD in pipeline
pwd | cat
When a built-in command is part of a pipeline, it executes in a child process (not the main shell process), so changes like cd or export don’t affect the shell environment.
Limitations and Edge Cases
Empty Commands
# Pipe with no command before or after
| grep error
# Output: syntax error: pipes
grep error |
# Output: syntax error: pipes
Multiple Consecutive Pipes
# Double pipe is an error
ls || grep
# Output: syntax error: pipes
# Triple pipe is an error
ls ||| grep
# Output: syntax error: pipes
Minishell does not support logical operators like || (OR) or && (AND). Only the pipe operator | for data flow is supported.
Pipe Before Redirection Operator
# Invalid syntax
ls | < input.txt grep error
# Output: syntax error: pipes
grep error | > output.txt
# Output: syntax error: redir
Buffering
Pipes use kernel buffers (typically 64KB):
# Small data - no blocking
echo "hello" | cat
# Large data - buffered automatically
cat large_file.txt | grep pattern
Process Overhead
Each command in a pipeline runs in a separate process:
# Creates 5 processes
cmd1 | cmd2 | cmd3 | cmd4 | cmd5
For long pipelines with many commands, there’s a small overhead from process creation and context switching, but this is usually negligible for interactive use.
Advanced Pipeline Patterns
Sorting and Uniqueness
# Find unique items
cat list.txt | sort | uniq
# Count occurrences
cat data.txt | sort | uniq -c | sort -rn
# Top 10 most frequent
cat access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn | head -10
Text Processing
# Extract specific fields
cat data.csv | cut -d',' -f2 | sort | uniq
# Count lines, words, characters
cat document.txt | wc
# Filter and transform
cat data.txt | grep "pattern" | tr '[:lower:]' '[:upper:]' | sort
System Monitoring
# Process monitoring
ps aux | grep -v grep | grep python | wc -l
# Disk usage
df -h | grep "/dev/" | awk '{print $5}' | sort -rn
# Network connections
netstat -tuln | grep LISTEN | wc -l
Exit Status with Pipes
The exit status of a pipeline is the exit status of the last command:
# Last command succeeds
ls /nonexistent | grep txt
echo $?
# Output: 0 (grep succeeded, even though ls failed)
# Last command fails
echo "hello" | grep "goodbye"
echo $?
# Output: 1 (grep found no match)
Minishell stores the exit status of the last command in the pipeline in the $? variable, following standard shell behavior.
Practical Use Cases
Log Analysis
# Find errors in logs
cat /var/log/app.log | grep ERROR | wc -l
# Get unique error types
cat app.log | grep ERROR | cut -d':' -f2 | sort | uniq
# Time-based filtering
cat access.log | grep "2024-03" | grep "404" | wc -l
File Management
# Find large files
ls -lh | sort -k5 -h | tail -10
# Count file types
ls | grep "\.." | rev | cut -d'.' -f1 | rev | sort | uniq -c
# Find old files
ls -lt | tail -20
Data Processing
# CSV processing
cat data.csv | grep "active" | cut -d',' -f1,3 | sort
# JSON processing (with jq)
cat data.json | jq '.users[]' | grep "email"
# Number crunching
cat numbers.txt | sort -n | tail -1
Implementation Details
For developers interested in the internals:
- Pipe counting: ft_count_pipes.c:92-106
- Pipeline execution: ft_execute_commands.c:68-95
- Process creation: Fork creates child for each command
- Pipe creation:
pipe() system call creates pipe pairs
- File descriptor management: dup2() redirects stdin/stdout
- Wait for completion: ft_execute_commands.c:92-94