SaltyCrane Blog — Notes on JavaScript and web development

Using psycopg2 with virtualenv on Ubuntu JauntyMaverick

Date:  |  Modified:  |  Tags: install_setup, python, ubuntu  |  23 Comments

Update 2009-11-02: Well I am dumb-- psycopg2 can be installed with pip/easy_install. The reason I got the error before was because I didn't have the required dependencies installed. On Ubuntu, I used apt-get build-dep. So, here's the summary:

Update 2009-11-11: My update doesn't work. See comments #4 and #5 below.

Update 2010-05-17: Here is what finally worked for me using Python 2.6 on Ubuntu 10.04 Lucid Lynx using virtualenv 1.4.8 and pip 0.7.1. Thanks to Daniel for the final piece of the solution.

Update 2011-05-13: Looks like the mx+virtualenv issue has been reported and fixed to be released in 2.4.2. via @psycopg's tweet

Update 2011-11-05: psycopg2 2.4.2 has been released with the mxDateTime fix so the install is very easy now.

Install dependencies

$ sudo apt-get build-dep python-psycopg2

Install pyscopg2 in a virtualenv

$ virtualenv --no-site-packages myenv 
$ source myenv/bin/activate 
$ pip install psycopg2

No longer needed: $ easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base



*** THE REST OF THIS POST DESCRIBES MY INITIAL OUTDATED SOLUTION. ***

I recently starting using virtualenv for creating isolated Python environments. Virtualenv has made it easy to manage different Python configurations for different websites, but I was slowed down a bit trying to use psycopg2, the Python-PostgreSQL database adapter, with virtualenv after upgrading to Ubuntu 9.04 Jaunty Jackalope.

Currently, virtualenv (1.3.3) doesn't find Ubuntu Jaunty's Python packages. In Ubuntu Jaunty, the default Python version changed from 2.5 to 2.6. More importantly, the site-packages directory was changed to dist-packages. Prior to Ubuntu 9.04, Ubuntu's Python packages, such as python-psycopg2, were installed to /usr/lib/python2.5/site-packages. Now Ubuntu's Python packages are installed to /usr/lib/python2.6/dist-packages. (See this discussion at the virtualenv group for more information.)

As a result of this change, virtualenv (as of 1.3.3) doesn't find Ubuntu's Python packages installed using apt-get. My solution was to create symlinks to the desired packages and egg-info files in site-packages. I'm not sure if this is the proper way to handle this. If there is a better solution, please let me know. One advantage of using this method is that I don't need to clutter my virtualenv with all the packages that have accumulated in my global site-packagesdist-packages.

Install easy_install, pip, and virtualenv

sudo apt-get install python-setuptools python-dev build-essential
sudo easy_install -U pip
sudo pip install -U virtualenv

Install Ubuntu's psycopg2 package

sudo apt-get install python-psycopg2

Symlink the psycopg2 (and mxDateTime) files

sudo mkdir /usr/lib/python2.6/site-packages
sudo ln -s /usr/lib/python2.6/dist-packages/psycopg2 /usr/lib/python2.6/site-packages
sudo ln -s /usr/lib/python2.6/dist-packages/psycopg2-2.0.8.egg-info /usr/lib/python2.6/site-packages
sudo ln -s /usr/lib/python2.6/dist-packages/mx /usr/lib/python2.6/site-packages

Create a virtualenv

virtualenv myenv

Check what's available

pip freeze -E myenv

Results:

psycopg2==2.0.8
wsgiref==0.1.2


Note: you might wonder why I didn't do a pip install -E myenv psycopg2. I tried this, but got an error. Maybe psycopg2 doesn't support pip/easy_install? Here is my error message:

Downloading/unpacking psycopg2
  Downloading psycopg2-2.0.11.tar.gz (255Kb): 255Kb downloaded
  Running setup.py egg_info for package psycopg2
    error: No such file or directory
    Complete output from command python setup.py egg_info:
    running egg_info

creating pip-egg-info/psycopg2.egg-info

writing pip-egg-info/psycopg2.egg-info/PKG-INFO

writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt

writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt

writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

warning: manifest_maker: standard file '-c' not found

error: No such file or directory

----------------------------------------
Command python setup.py egg_info failed with error code 1
Storing complete log in ./pip-log.txt
Complete output from command temp/bin/python /usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py install -E temp psycopg2 temp ___VENV_RESTART___:

----------------------------------------
Traceback (most recent call last):
  File "/usr/local/bin/pip", line 3, in 
    pip.main()
  File "/usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py", line 926, in main
    return command.main(initial_args, args[1:], options)
  File "/usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py", line 258, in main
    restart_in_venv(options.venv, site_packages, complete_args)
  File "/usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py", line 1009, in restart_in_venv
    call_subprocess([python, file] + args + [base, '___VENV_RESTART___'])
  File "/usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py", line 3643, in call_subprocess
    % (command_desc, proc.returncode))
pip.InstallationError: Command temp/bin/python /usr/local/lib/python2.6/dist-packages/pip-0.4-py2.6.egg/pip.py install -E temp psycopg2 temp ___VENV_RESTART___ failed with error code 1

Comments


#1 aw commented on :

Thanks, this worked for me.

Another way (but I haven't tried this) might be to install via setup.py from within the virtualenv:

(myenv)$ wget http://initd.org/pub/software/psycopg/psycopg2-2.0.11.tar.gz (myenv)$ tar zxf psycopg2-2.0.11.tar.gz (myenv)$ cd psycopg2-2.0.11 (myenv)$ python setup.py install running install running build [...] (myenv)$ python -c 'import psycopg2'


#2 aw commented on :

ugh, messed up the line breaks:

Another way (but I haven't tried this) might be to install via setup.py from within the virtualenv:

(myenv)$ wget http://initd.org/pub/software/psycopg/psycopg2-2.0.11.tar.gz
(myenv)$ tar zxf psycopg2-2.0.11.tar.gz
(myenv)$ cd psycopg2-2.0.11
(myenv)$ python setup.py install
running install
running build
[lots of messages...]

(myenv)$ python -c 'import psycopg2'


#3 bw commented on :

(myenv)$ ../bin/python setup.py install

;-)

..bw


#4 Shige Abe commented on :

This seemed to work fine for me as far as installing, but....

>>> import psycopg2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shige/bioaffinitycareers.com/lib/python2.6/site-packages/psycopg2/__init__.py", line 60, in <module>
from _psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: can't import mx.DateTime module

I googled it but nothing that was obvious to me helped.

Do you have any ideas?


#5 Eliot commented on :

Shige,
You are right. I got this same error. My update was not complete.

I know we need to install the mxDateTime library. But now I can't seem to install this with pip. I tried pip install -E myenv egenix-mx-base but got an error:

Downloading/unpacking egenix-mx-base
  Downloading egenix-mx-base-3.1.2.tar.gz (8.2Mb): 8.2Mb downloaded
  Running setup.py egg_info for package egenix-mx-base
    no build data file 'build/build-py2.6_ucs4.pck' found
Installing collected packages: egenix-mx-base
  Running setup.py install for egenix-mx-base
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: option --single-version-externally-managed not recognized
    Complete output from command /home/saltycrane/lib/python-environments/myenv/bin/python -c "import setuptools; __file__='/home/saltycrane/lib/python-environments/myenv/build/egenix-mx-base/setup.py'; execfile('/home/saltycrane/lib/python-environments/myenv/build/egenix-mx-base/setup.py')" install --single-version-externally-managed --record /tmp/pip-1VsR7f-record/install-record.txt --install-headers /home/saltycrane/lib/python-environments/myenv/lib/include:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]

   or: -c --help [cmd1 cmd2 ...]

   or: -c --help-commands

   or: -c cmd --help



error: option --single-version-externally-managed not recognized

----------------------------------------
Command /home/saltycrane/lib/python-environments/myenv/bin/python -c "import setuptools; __file__='/home/saltycrane/lib/python-environments/myenv/build/egenix-mx-base/setup.py'; execfile('/home/saltycrane/lib/python-environments/myenv/build/egenix-mx-base/setup.py')" install --single-version-externally-managed --record /tmp/pip-1VsR7f-record/install-record.txt --install-headers /home/saltycrane/lib/python-environments/myenv/lib/include failed with error code 1
Storing complete log in ./pip-log.txt

#6 Tim commented on :

This sequence of commands seemed to work for me (helped greatly by this post):

$ sudo apt-get build-dep python-psycopg2
$ virtualenv ENV
$ source ENV/bin/activate
$ easy_install psycopg2
$ easy_install egenix-mx-base

For some reason you can't install egenix with pip..


#7 Eliot commented on :

Tim, Thank you for figuring this out and posting your solution.


#8 Joebie commented on :

I had the same problem under fedora-12, for me the solution was installing gcc and the postgresql-devel packages.


#9 Ben commented on :

I had the exact same problem with psycopg2 in CentOS, but was able to solve it the same way Joebie did. I installed the development extras like so:

$ yum install postgres-devel

And everything seems to be working fine so far.

Thanks much for this post. I've just started a new project and resolved myself to try and start using tools like virtualenv this time around. I doubt I could get there without valuable help from leaders such as yourself.

Sincerely,

Ben.


#10 anothertrad commented on :

Hi folks! In my case, on Ubuntu, running python 2.5, I have created an environment without inherit the native libs of the system. I couldn't import psycopg2. Then I have downloaded the code, and when I try to build (both on my environment or out of it): Warning: Unable to find 'pg_config' filebuilding 'psycopg2._psycopg' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.0.14 (dt dec ext pq3)" -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -DHAVE_PQPROTOCOL3=1 -I/usr/include/python2.5 -I. -c psycopg/psycopgmodule.c -o build/temp.linux-i686-2.5/psycopg/psycopgmodule.o In file included from psycopg/psycopgmodule.c:32: ./psycopg/psycopg.h:31:22: error: libpq-fe.h: No such file or directory In file included from psycopg/psycopgmodule.c:33: ./psycopg/connection.h:72: error: expected specifier-qualifier-list before \u2018PGconn\u2019 ./psycopg/connection.h:94: error: expected declaration specifiers or \u2018...\u2019 before \u2018PGconn\u2019 In file included from psycopg/psycopgmodule.c:34: ./psycopg/cursor.h:61: error: expected specifier-qualifier-list before \u2018PGresult\u2019 In file included from psycopg/psycopgmodule.c:35: ./psycopg/lobject.h:31:28: error: libpq/libpq-fs.h: No such file or directory In file included from psycopg/psycopgmodule.c:35: ./psycopg/lobject.h:51: error: expected specifier-qualifier-list before \u2018Oid\u2019 ./psycopg/lobject.h:57: error: expected declaration specifiers or \u2018...\u2019 before \u2018Oid\u2019 ./psycopg/lobject.h:57: error: expected declaration specifiers or \u2018...\u2019 before \u2018Oid\u2019 psycopg/psycopgmodule.c: In function \u2018psyco_register_type\u2019: psycopg/psycopgmodule.c:254: error: \u2018cursorObject\u2019 has no member named \u2018string_types\u2019 psycopg/psycopgmodule.c:257: error: \u2018connectionObject\u2019 has no member named \u2018string_types\u2019 error: command 'gcc' failed with exit status 1

I have the latest build-essential. Any Ideas?


#11 anothertrad commented on :

Oh, I almost forgot: I already tried to:

0) sudo -s

1) apt-get remove python-psycopg2

2) enter in my virtual env

3) apt-get install python-psycopg2

but I can only import outside my environment, just like before


#12 Daniel commented on :

I had to do this:

easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base

To get it to work on ubuntu 10.04 + Python 2.6

cheers


#13 Ryan commented on :

I upgraded to 10.4 lynx and had a similar error, specifically...

error: libpq/libpq-fs.h: No such file or directory

I tried easy_install egenix-mx-base, but this did not work for me.

I had to install the 'libpq-dev' package which contains the Postgres header files required when building psycopg2.

This now works for me, thanks all.


#14 Eliot commented on :

Daniel: Thank you. Your solution was the one that worked for me on Ubuntu Lucid. I updated the post.

Ryan: I didn't need to do this, but maybe our configurations our different. Thanks for adding your notes.


#15 Mike commented on :

...succinctly getting me out of a tight spot.

I raise my cup of java in your honor.


#16 Pascal commented on :

Hi,
I'm on Ubuntu 10.04.
I created a virtualenv with -p python2.5 --no-site-packages
I did: apt-get install libpq-dev
In the virtualenv, I did: easy_install -i http://downloads.egenix.com/python/index/ucs4/ egenix-mx-base

But it doesn't work :-(

Installing collected packages: psycopg2
Running setup.py install for psycopg2
building 'psycopg2._psycopg' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.2.2 (dt dec ext pq3)" -DPG_VERSION_HEX=0x080404 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -DHAVE_PQPROTOCOL3=1 -I/usr/include/python2.5 -I. -I/usr/include/postgresql -I/usr/include/postgresql/8.4/server -c psycopg/psycopgmodule.c -o build/temp.linux-i686-2.5/psycopg/psycopgmodule.o -Wdeclaration-after-statement
psycopg/psycopgmodule.c:27:20: error: Python.h: No such file or directory
In file included from psycopg/psycopgmodule.c:31:
./psycopg/python.h:31:26: error: structmember.h: No such file or directory
./psycopg/python.h:34:4: error: #error "psycopg requires Python >= 2.4"
In file included from psycopg/psycopgmodule.c:32:

Any ideas how to solve that? Thanks.


#17 hunter commented on :

Thank you -your final solution helped me out! Your blog overall has been a great resource as well. I totally appreciate the efforts!


#18 w1sh commented on :

What's the purpose of the DateTime stuff anyway? Could I just leave that out or is it important?


#20 Chris commented on :

Your exact symlinking trick didn't work for me, but it did work when I symlinked psycopg2 directly into the site-packages inside my environment's /lib/python2.7/site-packages.


#21 Matty commented on :

Tim thanks. Your solution saved me some time.


#22 Marcus commented on :

I also had problems with easy_install in an activated virtualenv:

easy_install psycopg2

...didn't work

Error: pg_config executable not found.

Please add the directory containing pg_config >to the PATH or specify the full executable path with the >option:

python setup.py build_ext --pg-config >/path/to/pg_config build ...

or with the pg_config option in 'setup.cfg'. error: Setup script exited with 1

I suggets to sudo apt-get install python-psycopg2 ...and then manually link the /usr/python[version]/dist-packages to your [virtualenv]/lib/python[version]/site-packages...

sudo apt-get install python-psycopg2
cd [virtualenv]
source bin/activate
cd lib/python[version]/site-packages
ln -s /usr/lib/python[version]/dist-package/psycopg2 .
ln -s /usr/lib/python[version]/dist-package/psycopg2-2.0.13.egg-info .
ln -s /usr/lib/python[version]/dist-package/mx .

[vitualenv] is the path to your virtual environment [version] is the python version your using

Good luck


#23 Line2sun commented on :

Thanks! It was useful! :)