Bash 101 - I/O Redirection and Pipes

Bash scripting is a subject I'm really fond of, but before starting to write scripts one needs to understand how the bash commands work and how they can be connected. That's the purpose of this post, to provide some background before getting into the actual writing of scripts.

I/O Redirection

Many commands such as ls print their output on the display. We can change this using operators under the I/O redirection category, to redirect the output of those commands to files, devices and also to the input of other commands.

There are 3 file descriptors, stdin (standard input), stdout (standard output) and stderr (standard error). Basically you can use redirection in 7 different ways:

1. stdout 2 file
2. stderr 2 file
3. stdout 2 stderr
4. stderr 2 stdout
5. stderr and stdout 2 file
6. stderr and stdout 2 stdout
7. stderr and stdout 2 stderr

This file descriptors are represented by the numbers 0 (stdin), 1 (stdout) and 2 (stderr).
To make this clearer, let's give examples for each of the redirections.

1. ls -l > output.txt
2. grep something * 2> errors.txt
3. ls -l 1>&2
4. grep something * 2>&1
5. rm -f $(find / -name core) &> /dev/null (redirects both stdout and stderr to /dev/null that automatically deletes that information)

Other cases:

> filename - this truncates the file to zero length and if the file is not present has the same effect as touch

>> filename - this operator can be used as normal redirection to a file, but instead of writing from the start it appends to the end of that file

command < filename - accepts input from a file

3<> filename - open file for reading and writing and assigns file descriptor 3 to it

n <&- - closes file descriptor n (this is useful with pipes, because child processes inherit open file descriptors)


Another kind of redirection are the HERE documents that I've mentioned before. With those you can redirect a dynamic file to the input of a program that receives a normal file. An example of this can be seen in my earlier post.

Pipes

Very simply put, pipes let you use the output of a program as the input of another. To do this, a process is created for each command. That's why being careful with which file descriptors are open at the moment of creation (before sending the output as input) of each of those processes is important.

The basic functionality of pipes can be understood with a very simple example that has the same effect as ls -l *.sh:

ls -l  |  grep "\.sh$"

Here, the output of the program ls -l is sent to the grep program, which, in turn, will print lines which match the regex "\.txt$".