Python Script Output Redirection

“Write programs to handle text streams, because that is a universal interface.”

– Doug McIlroy, the inventor of Unix pipes (source)

Today, for the Nth time I got stuck while trying to redirect a Python script’s output to a file for later processing. So, after figuring it out again, I’m finally documenting my findings.

Let’s explore the default behavior for how Python scripts output to stdout vs stderr, and thus how we can save that output to a file or pipe it to another program (i.e. following the Unix Philosophy)

Example Script

Here’s a script that outputs a text using both the basic print command and the built-in logging module.

import logging
logging.basicConfig(level=logging.INFO, format="%(message)s")

print("printed message")
logging.info("log message")

As expected, running the script outputs each line to a terminal.

$ python script.py
printed message
log message

Python sends print output to stdout

Redirecting the scripts stdout only captures the print message, leaving the log messages in the terminal.

$ python script.py > stdout.log
log message

$ cat stdout.log
printed message

Python sends logging output to stderr

Redirecting stderr does the reverse, capturing the log output and leaving the print messages in the terminal.

$ python script.py 2> stderr.log
printed message

$ cat stderr.log
log message

How to redirect print and logging output to the same file

Both stdout and stderr can be redirected to the same file by appending the command with 2>&1 (see I/O redirection docs).

$ python script.py > output.log 2>&1

$ cat output.log
log message
printed message

I’m not sure why the print output appears after the logging output, when the script executes print before logging. If you know why, please comment below!

Piping output to another command

Another key principle in Unix-land is that the output of one program can be the input to another program. Connecting one program’s output to another program’s input is done with the pipe character | and is sometimes called “piping”.

We’ll start by using a command called grep which filters its input to only output the lines which contain a certain string.

In my shell, grep highlights the found string, which I will indicate here by adding *stars*

Default Pipe Behavior

$ python script.py | grep message
log message
printed *message*

Notice that only the print message was highlighted. By default, the pipe only redirects stdout, so grep only processed the print output. The logging messages went to stderr and thus were still output to the terminal even though they weren’t processed by grep.

Piping stdout and stderr

To send all output to grep we can use the same 2>&1 syntax from before.

$ python script.py 2>&1 | grep message
log *message*
printed *message*

Now grep is processing every line.

Piping only stderr

If you want grep to only process the stderr output (i.e. log outputs), we have to send the stdout somewhere else. On Linux we can choose /dev/null which is a text trashcan.

$ (python script.py 2>&1 1>/dev/null) | grep message 
log *message*

Notice that the print output was discarded, while everything else was processed by grep.