SaltyCrane Blog — Notes on JavaScript and web development

How to get stdout and stderr using Python's subprocess module

Date:  |  Modified:  |  Tags: processes, python  |  10 Comments

I wrote previously about how to get stdout and stderr using os.popen4. However, per the Python documentation, using the subprocess module is preferred:

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module intends to replace several other, older modules and functions, such as:

os.system
os.spawn*
os.popen*
popen2.*
commands.*

See the subprocess module documentation for more information.

Here is how to get stdout and stderr from a program using the subprocess module:

from subprocess import Popen, PIPE, STDOUT

cmd = 'ls /etc/fstab /etc/non-existent-file'
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = p.stdout.read()
print output

Results:

ls: cannot access /etc/non-existent-file: No such file or directory
/etc/fstab

Comments


#1 Ian McCracken commented on :

Here's a cleaner way:

from subprocess import Popen, PIPE
cmd = 'blah'
p = Popen(cmd, stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()

That way you wait for the output until the process has actually completed.

Also, take a look at the Process objects in the cliutils package. (Disclaimer: I wrote it.) They let you do some cool things.


#2 Eliot commented on :

Ian,
This looks like an interesting alternative. I just took my example from the Replacing os.popen* section in the docs.


#3 Flemmingbjerke commented on :

There is error here: p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)

Python says: NameError: name 'STDOUT' is not defined

If I use stderr=PIPE, it works.

This one is not good enough either: p = Popen(cmd, stdout=PIPE, stderr=PIPE)

It works fine with ls, rmdir etc. but with wget it seems to make errors whatever you do. But, the following seems to work fine with wget

p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)

Then, the following seems to work fine:

stdout, stderr = p.communicate()

Ian's p.communicate


#4 Ron commented on :

to Flemmingbjerke:

You got "'STDOUT' is not defined" because you must either: from subprocess import STDOUT or you must refer to STDOUT as subprocess.STDOUT .

I'm surprised you had any success running ls and rmdir without shell=True. ls and rmdir are not programs, they are internal commands within the shell program.

wget, however, is a program. You can use shell=True, but you don't need to. Using shell=False (the default) will save you from having to start a new shell process (which is an expensive operation). When you submit a command with shell=True, you submit the whole command as one string. But when you submit the command with shell=False, you must issue the command as a list of strings, with the command name in the first element of the list, the first argument in the next list element, etc. For example:

wgproc = subprocess.Popen(['wget', '-r', '--tries=10', 'http://fly.srk.fer.hr/', '-o', 'log'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
(standardout, junk) = wgproc.communicate()

#5 Pas commented on :

Hm. ls, rm, rmdir and others are programs. They live under /bin, just do a which ls.

And thanks for the useful post and comments :)


#6 Sebastian commented on :

Thanks for the hint on subprocess.STDOUT.


#7 billrie commented on :

Ha! Ron, you're an idiot.


#8 Komrad commented on :

How do you print the contents of stdout and stderr out in an easy to read fashion? if I do for line in stdout, it prints each character as separate line instead of printing each LINE on a separate line. Halp!

disqus:2097571561


#9 Eliot commented on :

Komrad, doing for line in p.stdout iterates over each line. See documentation here: https://docs.python.org/2/t... Make sure you're not calling read() like I am doing in the blog post. Do this:

    from subprocess import Popen, PIPE, STDOUT  

    cmd = 'ls /etc/fstab /etc/non-existent-file'  
    p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)  
    for line in p.stdout:  
        line = line.rstrip()  
        print line

disqus:2098478193


#10 Komrad commented on :

End ended up doing the (stdout, stderr ) = p.communicate() method, and then used the stdout.splitlines() function to get the output into a list of lines to loop through. that method worked for purposes.

disqus:2101701591