Linux input/output redirection

0. cheat-sheet

# print to stdout

echo hello

# print to stderr

echo hello >& 2

# redirect stdout to a file

curl http://www.google.ca > /root/foo.txt

# redirect stdout to a file (appending instead of replacing)

curl http://www.google.ca >> /root/foo.txt

# redirect stderr to a file

curl http://www.google.xx 2> /root/foo.txt

# redirect stdin to a file

cat < /etc/passwd

# redirect stdout and stderr to a file

curl http://www.google.xx > /root/foo.txt 2>& 1

# redirect stderr to stdout

curl http://www.google.xx 2>& 1

# redirect stdout to stderr

curl http://www.google.xx >& 2

# redirect cmd2's stdin to cmd1's stdout

cmd1 | cmd2

Notes:

• > is used to redirect to a file by its name
• >& is used to redirect a file by its file descriptor(see below)
• The following 4 lines are the same
• >foo.txt echo hello
• 1>foo.txt echo hello
• echo hello > foo.txt
• echo hello 1> foo.txt

Common mistakes:

• cat 1.txt | sort > 1.txt

The 1.txt will be empty as a result. Before "cat" and "sort" are running, the shell will open 1.txt and truncate it when preparing redirection.

1. File and File Descriptor

In Unix, everything is a file. For each process, every opened file is represented by a number starting from 0. This number is called File Descriptor (fd).

Unix guarantees the first opened file's fd is 0, the second opened file's fd is 1, and so on. When a file is closed, the corresponding fd number can be reused. 

A file can be opened multiple times in the same process, so it has multiple fds.

A child process inherits its parent process's fds.

2. Standard Input, Standard Output, Standard Error Output

By convention, "fd = 0" is also called standard input (stdin); "fd = 1" is called standard output (stdout); "fd = 2" is called standard error output (stderr).

Usually, a shell process opens /dev/tty three times, so stdin, stdout, and stderr are all pointing to the same file.

All child processes created by the shell inherit these 3 fds so that they can use the terminal as input/output.

3. Understanding Input/Output Redirection

By Unix philosophy, all applications are supposed to write their output to fd=1, read from fd=0, and print errors to fd=2.

As a result, by default, all those input/outputs are from/to /dev/tty.

But the new process created (fork(2)) by a shell is able to close those "0", "1", "2" fds, and reuse the numbers to open other files before executing the program (execve(2)).  This is where 'redirection' comes from.

To make it more clear, let's have a real example.

[root@localhost ~]# echo $$

9483

[root@localhost ~]# ls -l /proc/9483/fd

total 0

lrwx------. 1 root root 64 Sep 28 10:26 0 -> /dev/ttyS0

lrwx------. 1 root root 64 Sep 28 10:26 1 -> /dev/ttyS0

lrwx------. 1 root root 64 Sep 28 10:26 2 -> /dev/ttyS0

lrwx------. 1 root root 64 Sep 28 10:26 255 -> /dev/ttyS0

[root@localhost ~]# cat > /root/foo.txt &

[root@localhost ~]# ps -ef | grep cat

root      9814  9483  0 10:31 ttyS0    00:00:00 cat

[root@localhost ~]# ls -l /proc/9814/fd

total 0

lrwx------. 1 root root 64 Sep 28 10:32 0 -> /dev/ttyS0

l-wx------. 1 root root 64 Sep 28 10:32 1 -> /root/foo.txt

lrwx------. 1 root root 64 Sep 28 10:31 2 -> /dev/ttyS0

4. Understanding Pipe

Pipe is a special file  in memory and used  to connect two processes. The shell redirect the first cmd's fd=1 and the sencond cmd's fd=0 to the same pipe file.

[root@localhost ~]# cat | cat &

[1] 9832

[root@localhost ~]# ps -ef | grep cat

root      9831  9483  0 10:35 ttyS0    00:00:00 cat

root      9832  9483  0 10:35 ttyS0    00:00:00 cat

[root@localhost ~]# ls -l /proc/9831/fd

total 0

lrwx------. 1 root root 64 Sep 28 10:35 0 -> /dev/ttyS0

l-wx------. 1 root root 64 Sep 28 10:35 1 -> pipe:[31208]

lrwx------. 1 root root 64 Sep 28 10:35 2 -> /dev/ttyS0

[root@localhost ~]# ls -l /proc/9832/fd

total 0

lr-x------. 1 root root 64 Sep 28 10:35 0 -> pipe:[31208]

lrwx------. 1 root root 64 Sep 28 10:35 1 -> /dev/ttyS0

lrwx------. 1 root root 64 Sep 28 10:35 2 -> /dev/ttyS0

5. Conclusion

The famous UNIX philosophy: 

Programmers ensure their programs read from 0, write into 1, report errors to 2.

What files 0,1,2 really point to is determined by users when running the program.