Installing beanstalkd and pybeanstalk on Ubuntu

Nikolay pointed me at beanstalkd, a lightweight, message queue service partially inspired by the popular caching system, memcached. It features a blocking "reserve" call so workers don't need to poll for new jobs. However, some might miss the data persistence since the queue is stored in memory. beanstalkd has client libraries in Erlang, Perl, PHP, Python, and Ruby. Of course, I will use the Python version, pybeanstalk. Simon Willison also found beanstalkd interesting. (Credit to him for some of the words above.)

Update 2008-10-13: For a helpful example on using beanstalkd and pybeanstalk see Parand's beanstalkd tutorial.

Here is how I installed the beanstalkd server and pybeanstalk client on Ubuntu. I found no Ubuntu package for beanstalkd, so I installed from source.

Install beanstalkd

• Install prerequistes

  $ sudo apt-get install libevent1 libevent-dev

• Download

  $ cd ~/lib
  $ wget http://xph.us/software/beanstalkd/rel/beanstalkd-1.0.tar.gz

• Unpack

  $ tar zxvf beanstalkd-1.0.tar.gz

• Make

  $ cd beanstalkd-1.0
  $ make

• Print help

  $ ./beanstalkd -h

Install PyYAML

• Download

  $ wget http://pyyaml.org/download/pyyaml/PyYAML-3.06.tar.gz

• Unpack

  $ tar zxvf PyYAML-3.06.tar.gz

• Put PyYAML-3.06/lib/yaml somewhere on your Python path or run python setup.py.

Install pybeanstalk

• Download

  $ wget http://pybeanstalk.googlecode.com/files/pybeanstalk-0.11.1.tar.gz

• Unpack

  $ tar zxvf pybeanstalk-0.11.1.tar.gz

• Put pybeanstalk-0.11.1/beanstalk somewhere on your Python path or run python setup.py

Run beanstalkd server

• $ ~/lib/beanstalkd-1.0/beanstalkd -d -l 127.0.0.5 -p 11300

Run test client

• Create a file and run it:

  from beanstalk import serverconn
  from beanstalk import job
  
  SERVER = '127.0.0.5'
  PORT = 11300
  
  # setup connection
  connection = serverconn.ServerConn(SERVER, PORT)
  connection.job = job.Job
  
  # produce data
  for i in range(5):
      print 'put data: %d' % i
      data = job.Job(data=str(i), conn=connection)
      data.Queue()
  
  # consume data
  while True:
      j = connection.reserve()
      print 'got data: %s' % j.data
      j.Finish()
  

  Results:

  put data: 0
  put data: 1
  put data: 2
  put data: 3
  put data: 4
  got data: 0
  got data: 1
  got data: 2
  got data: 3
  got data: 4