I learned about git bisect from this Stack Overflow poll: What are your favorite git features or tricks? I thought it was so cool that I wanted to share an example.

After upgrading from Django 1.2.3 to Django 1.3, something broke on the website I was working on. To figure out what was wrong, I used git bisect to find the Django revision that introduced the relevant change. I cloned the Django github repo and pip install -e'd it into my virtualenv. Then I used git bisect as follows:

• Start git bisect

  $ git bisect start 
  

  $ git bisect bad 1.3 
  

  $ git bisect good 1.2.3 
  Bisecting: a merge base must be tested
    [618153bd3b047529f6cfb50917c88270f30e8ea8] Fixed #10843: the textile tests now pass against the latest textile library.
  

  Note: 1.3 and 1.2.3 are git tags in the Django git repo. Commit SHA1 hashes can be used instead.

• Start the development server in another terminal. Load the webpage in the browser. The page loads without error.

  [04/Aug/2011 19:13:20] "GET /my/webpage/ HTTP/1.1" 200 16279

• Mark the revision good

  $ git bisect good 
  Bisecting: 774 revisions left to test after this (roughly 10 steps)
  [39591cddccffdf3b66cfaaa60b95257daa4ef8c5] Fixed #14781 - Setting "CACHE_PREFIX" should be "CACHE_KEY_PREFIX". Thanks to adamv for report and patch.
  

• Reload the page. This time it throws a 500 error.

  [04/Aug/2011 19:13:59] "GET /my/webpage/ HTTP/1.1" 500 118861

• Mark the revision bad

  $ git bisect bad 
  Bisecting: 387 revisions left to test after this (roughly 9 steps)
  [6630bd24f45c7967c49b90adf82c63dd7d93e6f7] Fixed #13863 -- Corrected decimal separator in Swedish format module. Thanks, Ulf Urdén.
  

• Reload

  [04/Aug/2011 19:15:45] "GET /my/webpage/ HTTP/1.1" 500 117536

• Mark the revision bad

  $ git bisect bad 
  Bisecting: 193 revisions left to test after this (roughly 8 steps)
  [4d19282beb19b95e73aec119b40ed40456065a21] Migrated custom_methods doctests. Thanks to Alex Gaynor.
  

• Etc...

  [04/Aug/2011 19:16:17] "GET /my/webpage/ HTTP/1.1" 500 117536

  $ git bisect bad 
  Bisecting: 96 revisions left to test after this (roughly 7 steps)
  [ed63689b9e3be80c3af51bfc5d6805bcedbae9f0] Added file missing from r13590.
  

  [04/Aug/2011 19:16:46] "GET /my/webpage/ HTTP/1.1" 200 16279

  $ git bisect good 
  Bisecting: 48 revisions left to test after this (roughly 6 steps)
  [2eca0cd7f5fe6efd1f177724a948cdde7475f54c] Fixed #13754 - Add a note about a test client session property gotcha
  

  [04/Aug/2011 19:17:12] "GET /my/webpage/ HTTP/1.1" 200 16279

  $ git bisect good 
  Bisecting: 24 revisions left to test after this (roughly 5 steps)
  [d7e3c7bad40e5f66169b8b951d74c87103ece07d] Fixed #13095 -- `formfield_callback` keyword argument is now more sane and works with widgets defined in `ModelForm.Meta.widgets`.  Thanks, hvdklauw for bug report, vung for initial patch, and carljm for review.
  

  [04/Aug/2011 19:17:31] "GET /my/webpage/ HTTP/1.1" 200 16279

  $ git bisect good 
  [I forgot to copy/paste the console output for this one]
  

  [04/Aug/2011 19:17:54] "GET /my/webpage/ HTTP/1.1" 500 117536

  $ git bisect bad 
  Bisecting: 5 revisions left to test after this (roughly 3 steps)
  [622bf3b2199e5e7015edccb00f1ab694291ca121] Fixed #11905: Raise an error on model form creation if a non-existent field was listed in fields. Thanks ben and copelco.
  

  [04/Aug/2011 19:18:17] "GET /my/webpage/ HTTP/1.1" 500 117536

  $ git bisect bad 
  Bisecting: 2 revisions left to test after this (roughly 2 steps)
  [57427e9d758eb37b43f6b7804b24c6f47c2fa456] Fixed a test so that it actually tests what it's supposed to test.
  

  [04/Aug/2011 19:18:37] "GET /my/webpage/ HTTP/1.1" 200 16279

  $ git bisect good 
  Bisecting: 0 revisions left to test after this (roughly 1 step)
  [dcb12158881cdcc619de0ae6d3f6cf674a0d4abb] Better error message for calling get_next_by_* on unsaved models. Patch from Marc Fargas. Fixed #7435.
  

  [04/Aug/2011 19:19:07] "GET /my/webpage/ HTTP/1.1" 200 16279

• After marking the last commit, git bisect shows the first bad commit as 622bf3b2199e5e7015edccb00f1ab694291ca121

  $ git bisect good
  622bf3b2199e5e7015edccb00f1ab694291ca121 is the first bad commit
  commit 622bf3b2199e5e7015edccb00f1ab694291ca121
  Author: kmtracey <kmtracey>
  Date:   Sat Sep 11 01:39:16 2010 +0000
  
      Fixed #11905: Raise an error on model form creation if a non-existent field was listed in fields. Thanks ben and copelco.
      
      
      git-svn-id: http://code.djangoproject.com/svn/django/trunk@13739 bcc190cf-cafb-0310-a4f2-bffc1f526a37
  
  :040000 040000 87ec1601b970938b55382a5aa66db74716552c3d 2d418dfe8dd9f67fea408e022aa13838086ce33b M      django
  :040000 040000 054e6f42b3b467e47f295f8fda6cdc8a7f2c054c 52ba5265ba9d6458f23c8cb4a8a9bdcdbfc1f590 M      tests</kmtracey>
  

Note: checking for failed unit tests instead of checking for a 500 error in the browser is another (probably better) way to test with git bisect.

Note 2: The problem with my code was that I was adding extra fields to a model formset in the wrong way. I was trying to use add_fields on a base model formset instead of creating a custom Form with the extra field and specifying it in the factory. This Stack Overflow answer solved the problem for me.

Extra details

Here's how I installed Django from github

• Clone the Django repo and pip install -e it in my virtualenv

  $ git clone https://github.com/django/django.git django-github 
  $ cd django-github 
  $ workon myenv 
  $ pip install -e ./ 
  

• In another terminal run the Django development server

  $ cd /myproject 
  $ workon myenv 
  $ python manage.py runserver 
  

I'm reading a csv file with the Python csv module and could not find a setting to remove trailing whitespace. I found this setting, Dialect.skipinitialspace, but it I think it only applies to leading whitespace. Here's a one-liner to delete leading and trailing whitespace that worked for me.

import csv


reader = csv.DictReader(
    open('myfile.csv'),
    fieldnames=('myfield1', 'myfield1', 'myfield3'),
)

# skip the header row
next(reader)

# remove leading and trailing whitespace from all values
reader = (
    dict((k, v.strip()) for k, v in row.items() if v) for row in reader)

# print results
for row in reader:
    print row

Example run with lxml 2.3, Python 2.6.6 on Ubuntu 10.10

from lxml import objectify, etree

xml = '''
<dataset>
  <statusthing>success</statusthing>
  <datathing gabble="sent">[email protected]</datathing>
  <datathing gabble="not sent"></datathing>
</dataset>
'''

root = objectify.fromstring(xml)

print root.tag
print root.text
print root.attrib
# dataset
# None
# {}

print root.statusthing.tag
print root.statusthing.text
print root.statusthing.attrib
# statusthing
# success
# {}

for e in root.datathing:
    print e.tag
    print e.text
    print e.attrib
    print e.attrib['gabble']
# datathing
# [email protected]
# {'gabble': 'sent'}
# sent
# datathing
# None
# {'gabble': 'not sent'}
# not sent

for e in root.getchildren():
    print e.tag
# statusthing
# datathing
# datathing

for e in root.iterchildren():
    print e.tag
# statusthing
# datathing
# datathing

# you cannot modify the text attribute of an element.
# instead just assign to the element itself.
try:
    root.statusthing.text = 'failure'
except:
    import traceback
    traceback.print_exc()
# Traceback (most recent call last):
#   File "lxml_ex.py", line 54, in <module>
#     root.statusthing.text = 'failure'
#   File "lxml.objectify.pyx", line 237, in lxml.objectify.ObjectifiedElement.__setattr__ (src/lxml/lxml.objectify.c:2980)
# TypeError: attribute 'text' of 'StringElement' objects is not writable

# modify element text and write it out as xml again
root.statusthing = 'failure'
xml_new = etree.tostring(root, pretty_print=True)
print xml_new
# <dataset>
#   <statusthing xmlns:py="http://codespeak.net/lxml/objectify/pytype" py:pytype="str">failure</statusthing>
#   <datathing gabble="sent">[email protected]</datathing>
#   <datathing gabble="not sent">
# </datathing></dataset>

# Use deannotate() to get rid of 'py:pytype' information
objectify.deannotate(root, cleanup_namespaces=True)
xml_new = etree.tostring(root, pretty_print=True)
print xml_new
# <dataset>
#   <statusthing>failure</statusthing>
#   <datathing gabble="sent">[email protected]</datathing>
#   <datathing gabble="not sent">
# </datathing></dataset>

# Add a child element to the root
c = etree.Element("thisdoesntmatter")
c.tag = "thisdoesntmattereither"
c.text = "mytext"
c.attrib['myattr'] = 'myvalue'
root.newchild = c
objectify.deannotate(root, cleanup_namespaces=True)
xml_new = etree.tostring(root, pretty_print=True)
print xml_new
# <dataset>
#   <statusthing>failure</statusthing>
#   <datathing gabble="sent">[email protected]</datathing>
#   <datathing gabble="not sent">
#   <newchild myattr="myvalue">mytext</newchild>
# </datathing></dataset></module>

References:¶

• http://lxml.de/objectify.html
• http://infohost.nmt.edu/tcc/help/pubs/pylxml/web/index.html

The trace module causes Python to print lines of code as they are executed. I learned about trace via @brandon_rhodes's tweet.

Trace a Python program

python -m trace -t myprogram.py myargs 

Trace with a Django development server

From my experience, trace doesn't work with Django's auto-reloader. Use --noreload option

python -m trace -t manage.py runserver --noreload 

Tracing with more control

This article shows how to write custom functions that are passed to sys.settrace.

Django trace tool, django-trace

I wrote a Django management command that uses sys.settrace with other Django management commands. https://github.com/saltycrane/django-trace.

Install

• $ pip install -e git://github.com/saltycrane/django-trace.git#egg=django_trace 

• Add 'django_trace' to INSTALLED_APPS in settings.py

Usage

$ python manage.py trace --help 
Usage: manage.py trace [options] [command]

Use sys.settrace to trace code

Options:
  -v VERBOSITY, --verbosity=VERBOSITY
                        Verbosity level; 0=minimal output, 1=normal output,
                        2=verbose output, 3=very verbose output
  --settings=SETTINGS   The Python path to a settings module, e.g.
                        "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be
                        used.
  --pythonpath=PYTHONPATH
                        A directory to add to the Python path, e.g.
                        "/home/djangoprojects/myproject".
  --traceback           Print traceback on exception
  --include-builtins    Include builtin functions (default=False)
  --include-stdlib      Include standard library modules (default=False)
  --module-only         Display module names only (not lines of code)
  --calls-only          Display function calls only (not lines of code)
  --good=GOOD           Comma separated list of exact module names to match
  --bad=BAD             Comma separated list of exact module names to exclude
                        (takes precedence over --good and --good-regex)
  --good-regex=GOOD_REGEX
                        Regular expression of module to match
  --bad-regex=BAD_REGEX
                        Regular expression of module to exclude (takes
                        precedence over --good and --good-regex)
  --good-preset=GOOD_PRESET
                        A key in the GOOD_PRESETS setting
  --bad-preset=BAD_PRESET
                        A key in the BAD_PRESETS setting
  --version             show program's version number and exit
  -h, --help            show this help message and exit

or

$ python manage.py trace runserver 
01->django.core.management:128:     try:
01->django.core.management:129:         app_name = get_commands()[name]
02-->django.core.management:95:     if _commands is None:
02-->django.core.management:114:     return _commands
01->django.core.management:130:         if isinstance(app_name, BaseCommand):
01->django.core.management:134:             klass = load_command_class(app_name, name)
02-->django.core.management:69:     module = import_module('%s.management.commands.%s' % (app_name, name))
03--->django.utils.importlib:26:     if name.startswith('.'):
03--->django.utils.importlib:35:     __import__(name)
04---->django.contrib.staticfiles.management.commands.runserver:1: from optparse import make_option
04---->django.contrib.staticfiles.management.commands.runserver:3: from django.conf import settings
04---->django.contrib.staticfiles.management.commands.runserver:4: from django.core.management.commands.runserver import BaseRunserverCommand
05----->django.core.management.commands.runserver:1: from optparse import make_option
05----->django.core.management.commands.runserver:2: import os
05----->django.core.management.commands.runserver:3: import re
05----->django.core.management.commands.runserver:4: import sys
05----->django.core.management.commands.runserver:5: import socket
05----->django.core.management.commands.runserver:7: from django.core.management.base import BaseCommand, CommandError
05----->django.core.management.commands.runserver:8: from django.core.servers.basehttp import AdminMediaHandler, run, WSGIServerException, get_internal_wsgi_application
06------>django.core.servers.basehttp:8: """
06------>django.core.servers.basehttp:10: import os
06------>django.core.servers.basehttp:11: import socket
06------>django.core.servers.basehttp:12: import sys
06------>django.core.servers.basehttp:13: import traceback
...

or

$ python manage.py trace --bad=django,SocketServer --calls-only runserver 
01->wsgiref:23: """
01->wsgiref.simple_server:11: """
02-->BaseHTTPServer:18: """
03--->BaseHTTPServer:102: class HTTPServer(SocketServer.TCPServer):
03--->BaseHTTPServer:114: class BaseHTTPRequestHandler(SocketServer.StreamRequestHandler):
02-->wsgiref.handlers:1: """Base classes for server/gateway implementations"""
03--->wsgiref.util:1: """Miscellaneous WSGI-related Utilities"""
04---->wsgiref.util:11: class FileWrapper:
03--->wsgiref.headers:6: """
04---->wsgiref.headers:42: class Headers:
03--->wsgiref.handlers:43: class BaseHandler:
03--->wsgiref.handlers:371: class SimpleHandler(BaseHandler):
03--->wsgiref.handlers:412: class BaseCGIHandler(SimpleHandler):
03--->wsgiref.handlers:453: class CGIHandler(BaseCGIHandler):
02-->wsgiref.simple_server:26: class ServerHandler(SimpleHandler):
02-->wsgiref.simple_server:42: class WSGIServer(HTTPServer):
02-->wsgiref.simple_server:83: class WSGIRequestHandler(BaseHTTPRequestHandler):
01->contextlib:53: def contextmanager(func):
01->contextlib:53: def contextmanager(func):
Validating models...

0 errors found
Django version 1.4, using settings 'myproj.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
01->myproj.wsgi:15: """
01->wsgiref.simple_server:48:     def server_bind(self):
02-->BaseHTTPServer:106:     def server_bind(self):
02-->wsgiref.simple_server:53:     def setup_environ(self):
01->wsgiref.simple_server:53:     def setup_environ(self):
01->wsgiref.simple_server:66:     def set_app(self,application)

Trace decorator¶

I also wrote a decorator that traces code execution of the function it is decorating: https://github.com/saltycrane/trace-tools

Install

pip install -e git://github.com/saltycrane/trace-tools.git#egg=trace_tools 

Usage

from trace_tools.decorators import trace

@trace()
def some_function_to_trace(arg):
    do_something()

@trace(max_level=2)
def some_function_to_trace(arg):
    do_something()

@another_decorator
@trace(
    max_level=4,
    ignore=(
        'httplib', 'logging', 'ssl', 'email', 'encodings', 'gzip', 'urllib',
        'multiprocessing', 'django', 'cgi', 'requests', 'cookielib', 'base64',
        'slumber', 'zipfile', 'redis'))
def some_other_function():
    do_something_else()

@trace(max_level=10, calls_only=False, ignore=('debugtools', 'blessings', 'ipdb', 'IPython',), ignore_builtins=True, ignore_stdlib=True)
def process(self, content):
    do_stuff()

@trace(max_level=115, calls_only=True, ignore=(
        'suds.resolver',
        'suds.sudsobject',
        'suds.xsd',
        'debugtools', 'blessings', 'ipdb', 'IPython',),
       ignore_builtins=True, ignore_stdlib=True)
def process(self, content):
    do_stuff()

A few months ago, I learned that vc-annotate displays a nicely colorized git blame in Emacs. Today I learned that it is also interactive. I can cycle through revisions using p and n, open the file at a specified revision with f, view the log with l, and show a diff with d or changeset diff with D. (Unfortunately, the diff is ugly and not in color.)

vc-annotate is included with Emacs. To use, open a version controlled file, and type C-x v g or M-x vc-annotate

I am using Emacs 23.1 on Ubuntu Maverick

The -L option is the key. It allows curl to redirect to the next URL. Here's how to download a tarball from github and untar it inline:

$ curl -L https://github.com/pinard/Pymacs/tarball/v0.24-beta2 | tar zx 

Via http://support.github.com/discussions/repos/1789-you-cant-download-a-tarball-with-curl

Alternatively, using wget:

$ wget --no-check-certificate https://github.com/pinard/Pymacs/tarball/v0.24-beta2 -O - | tar xz 

Inspired by articles such as Why you should learn just a little Awk and Learn one sed command, I am trying to make use of Unix tools sed, awk, grep, cut, uniq, sort, etc. instead of writing short Python utility scripts.

Here is a Python script I wrote this week. It greps a file for a given regular expression pattern and returns a unique, sorted, list of matches inside the capturing parentheses.

# grep2.py

import re
import sys


def main():
    patt = sys.argv[1]
    filename = sys.argv[2]

    text = open(filename).read()
    matchlist = set(m.group(1) for m in re.finditer(patt, text, re.MULTILINE))
    for m in sorted(matchlist):
        print m


if __name__ == '__main__':
    main()

As an example, I used my script to search one of the Django admin template files for all the Django template markup in the file.

$ python grep2.py '({{[^{}]+}}|{%[^{}]+%})' tabular.html 

Output:

{% admin_media_prefix %}
{% blocktrans with inline_admin_formset.opts.verbose_name|title as verbose_name %}
{% cycle "row1" "row2" %}
{% else %}
{% endblocktrans %}
{% endfor %}
{% endif %}
{% endspaceless %}
{% for field in inline_admin_formset.fields %}
{% for field in line %}
{% for fieldset in inline_admin_form %}
{% for inline_admin_form in inline_admin_formset %}
{% for line in fieldset %}
{% if field.is_hidden %}
{% if field.is_readonly %}
{% if field.required %}
{% if forloop.first %}
{% if forloop.last %}
{% if inline_admin_form.form.non_field_errors %}
{% if inline_admin_form.has_auto_field %}
{% if inline_admin_form.original %}
{% if inline_admin_form.original or inline_admin_form.show_url %}
{% if inline_admin_form.show_url %}
{% if inline_admin_formset.formset.can_delete %}
{% if not field.widget.is_hidden %}
{% if not forloop.last %}
{% load i18n adminmedia admin_modify %}
{% spaceless %}
{% trans "Delete?" %}
{% trans "Remove" %}
{% trans "View on site" %}
{{ field.contents }}
{{ field.field }}
{{ field.field.errors.as_ul }}
{{ field.field.name }}
{{ field.label|capfirst }}
{{ forloop.counter0 }}
{{ inline_admin_form.deletion_field.field }}
{{ inline_admin_form.fk_field.field }}
{{ inline_admin_form.form.non_field_errors }}
{{ inline_admin_form.original }}
{{ inline_admin_form.original.id }}
{{ inline_admin_form.original_content_type_id }}
{{ inline_admin_form.pk_field.field }}
{{ inline_admin_formset.formset.management_form }}
{{ inline_admin_formset.formset.non_form_errors }}
{{ inline_admin_formset.formset.prefix }}
{{ inline_admin_formset.opts.verbose_name_plural|capfirst }}
{{ inline_admin_form|cell_count }}
{{ verbose_name }}

Here's my attempt at using Unix tools:

$ sed -rn 's/^.*(\{\{.*\}\}|\{%.*%\}).*$/\1/gp' tabular.html | sort | uniq 

However the output isn't quite the same:

{% admin_media_prefix %}
{% else %}
{% endblocktrans %}
{% endfor %}
{% endif %}
{% endspaceless %}
{% for field in inline_admin_formset.fields %}
{% for field in line %}
{% for fieldset in inline_admin_form %}
{% for inline_admin_form in inline_admin_formset %}
{% for line in fieldset %}
{% if field.is_readonly %}
{% if inline_admin_form.form.non_field_errors %}
{% if inline_admin_form.original or inline_admin_form.show_url %}
{% if inline_admin_formset.formset.can_delete %}
{% if not field.widget.is_hidden %}
{% load i18n adminmedia admin_modify %}
{% spaceless %}
{% trans "Remove" %}
{{ field.contents }}
{{ field.field }}
{{ field.field.errors.as_ul }}
{{ field.field.name }}
{{ field.label|capfirst }}
{{ inline_admin_form.fk_field.field }}
{{ inline_admin_form.form.non_field_errors }}
{{ inline_admin_formset.formset.management_form }}
{{ inline_admin_formset.formset.non_form_errors }}
{{ inline_admin_formset.formset.prefix }}
{{ inline_admin_formset.opts.verbose_name_plural|capfirst }}

Unix tools are powerful and concise, but I still need to get a lot more comfortable with their syntax. Please leave a comment if you know how to fix my command.

By default, running subprocess.Popen with shell=True uses /bin/sh as the shell. If you want to change the shell to /bin/bash, set the executable keyword argument to /bin/bash.

Solution thanks this great article: Working with Python subprocess - Shells, Processes, Streams, Pipes, Redirects and More

import subprocess

def bash_command(cmd):
    subprocess.Popen(cmd, shell=True, executable='/bin/bash')

bash_command('a="Apples and oranges" && echo "${a/oranges/grapes}"')

Output:

Apples and grapes

For some reason, the above didn't work for my specific case, so I had to use the following instead:

import subprocess

def bash_command(cmd):
    subprocess.Popen(['/bin/bash', '-c', cmd])

See also

• Python subprocess.Popen documentation
• Ubuntu documentation on differences between dash and bash
• GNU Bash Reference Manual

Three years ago, I decided to buy a Dell Ubuntu desktop PC instead of building it myself. This time around, I went the other way. My main reasons were: better parts and easier upgrades. We'll see what I decide in 2014.

I used this guide: Build a Linux Media Center PC August 6, 2010

See also: 8 Reasons to Build Your Own PC