SaltyCrane Blog — Notes on JavaScript and web development

Django Blog Project #14: Running Django 1.0

I'm now running Django 1.0 for this blog. For those who haven't heard, Django 1.0 Final was released yesterday sometime. My last Django update had been shortly after Beta 2 was released and updating to 1.0 Final didn't require any code changes for me. (I think everything since then was supposed to be bug fixes.) (I also updated my work project to 1.0 with no problem.) That's about all the content for this post. Here are some links:

Here are my notes on my path to 1.0:


And here are some Django 1.0 links:


Django Blog Project #13: Updating Django 1.0 Beta 2 New Comments and Adding Markdown support

I've updated to Django 1.0 Beta 2. One of the big items for this release was the new commenting framework. I had been waiting for this, so I was excited to see it was finally done.

I also added support for Markdown formatting of my comments. I actually could have added this earlier, but I only recently learned that Django has built-in support for Markdown.

Update URLConf

When I glanced over the changes for the new commenting framework, I missed this change and I actually had to Google on my error message. Luckily, someone (I don't remember where I found it now) had run into the same problem and saved me.

~/src/django/myblogsite/urls.py:
--- a/urls.py   Thu Aug 21 10:05:20 2008 -0500
+++ b/urls.py   Mon Sep 01 22:34:16 2008 -0700
@@ -1,6 +1,6 @@
 from django.conf.urls.defaults import *
 from django.contrib import admin
-from django.contrib.comments.models import FreeComment
+from django.contrib.comments.models import Comment
 from iwiwdsmi.myblogapp.views import *
 from iwiwdsmi.feeds import *
 from iwiwdsmi.views import *
@@ -19,7 +19,7 @@
     (r'^admin/(.*)', admin.site.root),
     (r'^feeds/(?P<url>.*)/$', 'django.contrib.syndication.views.feed', 
      {'feed_dict': feeds}),
-    (r'^comments/', include('django.contrib.comments.urls.comments')),
+    (r'^comments/', include('django.contrib.comments.urls')),
     
     (r'^$', rootview),</url>
Update the database

See the Django Upgrading from Django's previous comment system guide for more complete information.

  • I ran
    $ cd ~/src/django/myblogsite
    $ python manage.py syncdb
  • Then I entered my sqlite3 shell:
    $ sqlite3 mydatabase.sqlite3
    and pasted the following at the command prompt:
    BEGIN;
    
    INSERT INTO django_comments
        (content_type_id, object_pk, site_id, user_name, user_email, user_url,
        comment, submit_date, ip_address, is_public, is_removed)
    SELECT
        content_type_id, object_id, site_id, person_name, '', '', comment,
        submit_date, ip_address, is_public, approved
    FROM comments_freecomment;
    
    INSERT INTO django_comments
        (content_type_id, object_pk, site_id, user_id, user_name, user_email,
        user_url, comment, submit_date, ip_address, is_public, is_removed)
    SELECT
        content_type_id, object_id, site_id, user_id, '', '', '', comment,
        submit_date, ip_address, is_public, is_removed
    FROM comments_comment;
    
    UPDATE django_comments SET user_name = (
        SELECT username FROM auth_user
        WHERE django_comments.user_id = auth_user.id
    ) WHERE django_comments.user_id is not NULL;
    UPDATE django_comments SET user_email = (
        SELECT email FROM auth_user
        WHERE django_comments.user_id = auth_user.id
    ) WHERE django_comments.user_id is not NULL;
    
    COMMIT;
    then exited:
    .exit
Templates

The rest of the changes were with the templates.

  • I removed my old comments templates:
    rm -rf ~/src/django/myblogsite/templates/comments
  • I copied the new templates:
    cp -r ~/lib/django_trunk/django/contrib/comments ~/src/django/myblogsite/templates
  • I updated ~/src/django/myblogsite/templates/listpage.html:
    --- a/templates/listpage.html   Thu Aug 21 10:05:20 2008 -0500
    +++ b/templates/listpage.html   Mon Sep 01 22:46:34 2008 -0700
    @@ -47,7 +47,7 @@
           {% endfor %}
           &nbsp;&nbsp;|&nbsp;&nbsp;
     
    -      {% get_free_comment_count for myblogapp.post post.id as comment_count %}
    +      {% get_comment_count for myblogapp.post post.id as comment_count %}
           <a href="{{ post.get_absolute_url }}#comments">
             {{ comment_count|add:post.lc_count }} 
             Comment{{ comment_count|add:post.lc_count|pluralize}}</a>
    
  • I updated ~/src/django/myblogsite/templates/singlepost.html:
    --- a/templates/singlepost.html Thu Aug 21 10:05:20 2008 -0500
    +++ b/templates/singlepost.html Tue Sep 02 00:44:51 2008 -0700
    @@ -1,6 +1,7 @@
     {% extends "base.html" %}
     
     {% load comments %}
    +{% load markup %}
     
     {% block title %}
       {{ main_title }}: {{ post.title }}
    @@ -59,8 +60,8 @@
       {% endfor %}
       <br />
     
    -  {% get_free_comment_list for myblogapp.post post.id as comment_list %}
    -  {% get_free_comment_count for myblogapp.post post.id as comment_count %}
    +  {% get_comment_list for myblogapp.post post.id as comment_list %}
    +  {% get_comment_count for myblogapp.post post.id as comment_count %}
       {% if comment_list %}
         <h4>{{ comment_count }} 
         {% if lc_list %}New {% endif %}
    @@ -69,13 +70,19 @@
       {% for comment in comment_list %}
         <br />
         <a name="c{{ comment.id }}" href="#c{{ comment.id }}">#{{ forloop.counter }}</a>
    -    <b>{{ comment.person_name|escape }}</b> commented, 
    -      on {{ comment.submit_date|date:"F j, Y" }} at {{ comment.submit_date|date:"P" }}:
    -    {{ comment.comment|escape|urlizetrunc:40|linebreaks }}
    +    <b>
    +      {% if comment.url %}
    +        <a href="{{ comment.url }}">{{ comment.name|escape }}</a>
    +      {% else %}
    +        {{ comment.name|escape }}
    +      {% endif %}
    +    </b> commented, 
    +    on {{ comment.submit_date|date:"F j, Y" }} at {{ comment.submit_date|date:"P" }}:
    +    {{ comment.comment|markdown:"safe" }}
       {% endfor %}
       <br />
     
       </h4><h4>Post a comment</h4>
    -  {% free_comment_form for myblogapp.post post.id %}
    +  {% render_comment_form for post %}
     
     {% endblock %}
    
Add django.contrib.markup to INSTALLED_APPS

To use Markdown, I added django.contrib.markup to my INSTALLED_APPS in settings.py ~/src/django/myblogsite/settings.py:

 INSTALLED_APPS = (
     'django.contrib.auth',
     'django.contrib.contenttypes',
     'django.contrib.sessions',
     'django.contrib.sites',
     'django.contrib.admin',
     'django.contrib.comments',
+    'django.contrib.markup',
     'iwiwdsmi.myblogapp',
 )

That was about it. I messed with the templates a little to try to make things a little prettier. I'm not completely satisfied yet though. My next step is to add django-openid support. Later on, I'd also like to add email notification and spam filtering.

Error messages:
got an unexpected keyword argument 'core'
Validating models...
Unhandled exception in thread started by 
Traceback (most recent call last):
  File "/home/sofeng/lib/python-packages/django/core/management/commands/runserver.py", line 47, in inner_run
    self.validate(display_num_errors=True)
  File "/home/sofeng/lib/python-packages/django/core/management/base.py", line 122, in validate
    num_errors = get_validation_errors(s, app)
  File "/home/sofeng/lib/python-packages/django/core/management/validation.py", line 28, in get_validation_errors
    for (app_name, error) in get_app_errors().items():
  File "/home/sofeng/lib/python-packages/django/db/models/loading.py", line 128, in get_app_errors  
    self._populate()
  File "/home/sofeng/lib/python-packages/django/db/models/loading.py", line 57, in _populate
    self.load_app(app_name, True)
  File "/home/sofeng/lib/python-packages/django/db/models/loading.py", line 72, in load_app
    mod = __import__(app_name, {}, {}, ['models'])
  File "/home/sofeng/src/django/mozblog/myblogapp/models.py", line 30, in 
    class LegacyComment(models.Model):
  File "/home/sofeng/src/django/mozblog/myblogapp/models.py", line 32, in LegacyComment
    website = models.URLField(core=False)
  File "/home/sofeng/lib/python-packages/django/db/models/fields/__init__.py", line 828, in __init__
    CharField.__init__(self, verbose_name, name, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'core'

I removed the core argument from my models. This is an oldforms related thing that has been removed. See here

Error while importing URLconf myblogsite.urls': cannot import name FreeComment

Comments have been refactored. See the Upgrading Guide

Notes on Django and MySql on Amazon's EC2

Install Elasticfox

Install the Elasticfox Firefox Extension for Amazon EC2: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609

Set up Amazon EC2 accounts and Elasticfox

Follow Arope's instructions for setting up Amazon EC2 accounts and Elasticfox. I used the alestic/ubuntu-8.04-hardy-base-20080628.manifest.xml machine image.

view standard apache page

In Elasticfox, right-click on your running instance and select "Copy Public DNS Name to clipboard". Then, paste that address in your browser. You should see Apache's "It works!" page.

ssh into instance

In Elasticfox, right-click on your running instance and select "SSH to Public Domain Name"

install stuff

Ubuntu Hardy has the following versions:

  • Apache 2.2.8
  • Mod_python 3.3.1
  • MySql 5.0.51
  • Django 0.96.1

On your remote instance, do the following.

# apt-get update
# apt-get install python-django
# apt-get install mysql-server
# apt-get install python-mysqldb
# apt-get install libapache2-mod-python

Update 2008-09-09: The Django mod_python documentation recommends using Apache's prefork MPM as opposed to the worker MPM. The worker MPM was installed by default on my Alestic Ubuntu image so I uninstalled it and replaced it with the prefork version.

# apt-get autoremove --purge apache2-mpm-worker
# apt-get install apache2-mpm-prefork

To see your current version of Apache, run the command: apache2 -V

create a django project
# cd /srv
# django-admin startproject mysite
configure django mod_python

See also Jeff Baier's article: Installing Django on an Ubuntu Linux Server for more information.

Edit /etc/apache2/httpd.conf and insert the following:

<location "/">
    SetHandler python-program
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE mysite.settings
    PythonPath "['/srv'] + sys.path"
    PythonDebug On
</location>
restart the apache server
# /etc/init.d/apache2 restart

You should see Django's "It Worked!" page.

Set up a MySql database and user

Note, use the password you entered when installing MySql

# mysql -u root -p
Enter password: 

mysql> CREATE DATABASE django_db;
Query OK, 1 row affected (0.01 sec)

mysql> GRANT ALL ON django_db.* TO 'djangouser'@'localhost' IDENTIFIED BY 'yourpassword';
Query OK, 0 rows affected (0.03 sec)

mysql> quit
Bye
Edit the Django database settings
Edit mysite/settings.py:
DATABASE_ENGINE = 'mysql'           # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' or 'ado_mssql'.
DATABASE_NAME = 'django_db'             # Or path to database file if using sqlite3.
DATABASE_USER = 'djangouser'             # Not used with sqlite3.
DATABASE_PASSWORD = 'yourpassword'         # Not used with sqlite3.
DATABASE_HOST = ''             # Set to empty string for localhost. Not used with sqlite3.
DATABASE_PORT = ''             # Set to empty string for default. Not used with sqlite3.
Do a 'syncdb' to create the database tables
# cd mysite
# python manage.py syncdb
Creating table auth_message
Creating table auth_group
Creating table auth_user
Creating table auth_permission
Creating table django_content_type
Creating table django_session
Creating table django_site

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (Leave blank to use 'sofeng'):    
E-mail address: [email protected]
Password: 
Password (again): 
Superuser created successfully.
Installing index for auth.Message model
Installing index for auth.Permission model
Loading 'initial_data' fixtures...
No fixtures found.
upload a mercurial django project

on the remote instance, install mercurial:

# apt-get install mercurial

on your local machine with the mercurial repo, run:

$ hg clone -e 'ssh -i /home/sofeng/.ec2-elasticfox/id_django-keypair.pem' yourproj ssh://[email protected]//srv/yourproj
where /home/sofeng/.ec2-elasticfox/id_django-keypair.pem is the private key associated with your instance and yourdns.compute-1.amazonaws.com is the public domain name associated with your instance.

back on the remote instance:

# cd /srv/mozblog
# hg update
# python manage.py syncdb
set up apache to serve static files
  • Create a link to the media files:
    # cd /var/www
    # ln -s /srv/mozblog/media site_media
    # ln -s /usr/share/python-support/python-django/django/contrib/admin/media/ admin_media
  • Edit /etc/apache2/httpd.conf:
    <location "/">
        SetHandler python-program
        PythonHandler django.core.handlers.modpython
        SetEnv DJANGO_SETTINGS_MODULE mozblog.settings
        PythonPath "['/srv'] + sys.path"
        PythonDebug On
    </location>
    <location "/site_media">
    
        SetHandler None
    </location>
    <location "/admin_media">
        SetHandler None
    </location>
Restart the apache server
# /etc/init.d/apache2 restart

How to conditionally replace items in a list

I wanted to replace items in a list based on a specific condition. For example, given a list of numbers, I want to replace all items that are negative with zero.

Naive way

At first, I thought of something like this:

mylist = [111, -222, 333, -444]
newlist = []
for item in mylist:
    if item < 0:
        item = 0
    newlist.append(item)
mylist = newlist
print mylist

Which gave me the expected results:

[111, 0, 333, 0]
Better way?

Then I tried using Python's enumerate (see my previous example) built-in function to replace the item in-line. This seems to be a more elegant solution to me. Is there a better way? How would you do it?

mylist = [111, -222, 333, -444]
for (i, item) in enumerate(mylist):
    if item < 0:
        mylist[i] = 0
print mylist

Results:

[111, 0, 333, 0]

Django Blog Project #12: Adding Pygments syntax highlighting

I've finally added automatic code highlighting to my blog. It uses Pygments to do the syntax highlighting and Beautiful Soup to find all the <pre> blocks to highlight. I still write my blog posts in HTML, but now add a class attribute to my <pre> tags to specify the Pygments lexer to use. For example, for python code, I use:

<pre class="python">
import this
def demo():
    pass</pre>

Which turns into:

import this
def demo():
    pass

I bought James Bennett's book, Practical Django Projects about a month ago and it has good information about creating a blog with Django. It also documented techniques for syntax highlighting which I used here. To summarize, I added a new attribute, called body_highlighted to my Post model. Then, I added a custom save() method which parses my original HTML with Beautiful Soup and highlights it with Pygments.

Model changes

Here is the relevant code in ~/src/django/myblogsite/myblogapp/models.py:

class Post(models.Model):
    # ...
    body = models.TextField()
    body_highlighted = models.TextField(editable=False, blank=True)

    def save(self):
        self.body_highlighted = self.highlight_code(self.body)
        super(Post, self).save()

    def highlight_code(self, html):
        soup = BeautifulSoup(html)
        preblocks = soup.findAll('pre')
        for pre in preblocks:
            if pre.has_key('class'):
                try:
                    code = ''.join([unicode(item) for item in pre.contents])
                    code = self.unescape_html(code)
                    lexer = lexers.get_lexer_by_name(pre['class'])
                    formatter = formatters.HtmlFormatter()
                    code_hl = highlight(code, lexer, formatter)
                    pre.replaceWith(BeautifulSoup(code_hl))
                except:
                    pass
        return unicode(soup)

    def unescape_html(self, html):
        html = html.replace('&lt;', '<')
        html = html.replace('&gt;', '>')
        html = html.replace('&amp;', '&')
        return html

Update 2010-04-09: I added the unescape_html method so that I could highlight Python code with regular expression named groups. For example:

m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds")
With the new fix in place, I just need to escape the < and > characters with &lt; and &gt; and the syntax highlighting will display correctly. Before I made the fix, if I did not escape the characters, BeautifulSoup would add closing tags to what it thought was my malformed HTML. So instead of the above, it looked like this:
m = re.match(r"(?P<first_name>\w+) (?P<last_name>\w+)", "Malcolm Reynolds")</last_name></first_name>
If anyone knows of a better solution, please let me know.

Update the database
  • List the SQL commands Django would use:
    $ cd ~/src/django/myblogsite/
    $ python manage.py sqlall myblogapp
    
    BEGIN;
    CREATE TABLE "myblogapp_post" (
        "id" integer NOT NULL PRIMARY KEY,
        "author_id" integer NOT NULL REFERENCES "auth_user" ("id"),
        "title" varchar(200) NOT NULL,
        "slug" varchar(200) NOT NULL,
        "date_created" datetime NOT NULL,
        "date_modified" datetime NOT NULL,
        "tags" varchar(200) NOT NULL,
        "body" text NOT NULL,
        "body_highlighted" text NOT NULL,
    )
    ;
    CREATE INDEX "myblogapp_post_author_id" ON "myblogapp_post" ("author_id");
    CREATE INDEX "myblogapp_post_slug" ON "myblogapp_post" ("slug");
    COMMIT;
  • Enter the sqlite3 shell:
    $ sqlite3 mydatabase.sqlite3
    

    and enter the following statements:
    sqlite> ALTER TABLE myblogapp_post ADD COLUMN body_highlighted text;
    sqlite> .exit
Update the template

Here is the relevant code in ~/src/django/myblogsite/templates/singlepost.html:

    {% if post.body_highlighted %}
      {{ post.body_highlighted|safe }}
    {% else %}
      {{ post.body|safe }}
    {% endif %}
Add CSS for Pygments

One last step is to add the CSS for Pygments. Here is an excerpt from my ~/src/django/myblogsite/media/css/mystyle.css:

/* PYGMENTS STYLE */
/* customized */
.c  { color: #008040; font-style: italic } /* Comment */
.cm { color: #008040; font-style: italic } /* Comment.Multiline */
.cp { color: #BC7A00 } /* Comment.Preproc */
.c1 { color: #008040; font-style: italic } /* Comment.Single */
.cs { color: #008040; font-style: italic } /* Comment.Special */
.gd { color: grey; text-decoration: line-through } /* Generic.Deleted */
.gi { color: red; } /* Generic.Inserted */
.k  { color: #000080; font-weight: bold } /* Keyword */
.kc { color: #000000; font-weight: bold } /* Keyword.Constant */
.kd { color: #000000; font-weight: bold } /* Keyword.Declaration */
.kp { color: #000000 } /* Keyword.Pseudo */
.kr { color: #000000; font-weight: bold } /* Keyword.Reserved */
.kt { color: #000000; font-weight: bold } /* Keyword.Type */

/* original settings */
.err { border: 1px solid #FF0000 } /* Error */
.o { color: #666666 } /* Operator */
.ge { font-style: italic } /* Generic.Emph */
.gr { color: #FF0000 } /* Generic.Error */
.gh { color: #000080; font-weight: bold } /* Generic.Heading */
.go { color: #808080 } /* Generic.Output */
.gp { color: #000080; font-weight: bold } /* Generic.Prompt */
.gs { font-weight: bold } /* Generic.Strong */
.gu { color: #800080; font-weight: bold } /* Generic.Subheading */
.gt { color: #0040D0 } /* Generic.Traceback */
.m { color: #666666 } /* Literal.Number */
.s { color: #BA2121 } /* Literal.String */
.na { color: #7D9029 } /* Name.Attribute */
.nb { color: #008000 } /* Name.Builtin */
.nc { color: #0000FF; font-weight: bold } /* Name.Class */
.no { color: #880000 } /* Name.Constant */
.nd { color: #AA22FF } /* Name.Decorator */
.ni { color: #999999; font-weight: bold } /* Name.Entity */
.ne { color: #D2413A; font-weight: bold } /* Name.Exception */
.nf { color: #0000FF } /* Name.Function */
.nl { color: #A0A000 } /* Name.Label */
.nn { color: #0000FF; font-weight: bold } /* Name.Namespace */
.nt { color: #008000; font-weight: bold } /* Name.Tag */
.nv { color: #19177C } /* Name.Variable */
.ow { color: #AA22FF; font-weight: bold } /* Operator.Word */
.w { color: #bbbbbb } /* Text.Whitespace */
.mf { color: #666666 } /* Literal.Number.Float */
.mh { color: #666666 } /* Literal.Number.Hex */
.mi { color: #666666 } /* Literal.Number.Integer */
.mo { color: #666666 } /* Literal.Number.Oct */
.sb { color: #BA2121 } /* Literal.String.Backtick */
.sc { color: #BA2121 } /* Literal.String.Char */
.sd { color: #BA2121; font-style: italic } /* Literal.String.Doc */
.s2 { color: #BA2121 } /* Literal.String.Double */
.se { color: #BB6622; font-weight: bold } /* Literal.String.Escape */
.sh { color: #BA2121 } /* Literal.String.Heredoc */
.si { color: #BB6688; font-weight: bold } /* Literal.String.Interpol */
.sx { color: #008000 } /* Literal.String.Other */
.sr { color: #BB6688 } /* Literal.String.Regex */
.s1 { color: #BA2121 } /* Literal.String.Single */
.ss { color: #19177C } /* Literal.String.Symbol */
.bp { color: #008000 } /* Name.Builtin.Pseudo */
.vc { color: #19177C } /* Name.Variable.Class */
.vg { color: #19177C } /* Name.Variable.Global */
.vi { color: #19177C } /* Name.Variable.Instance */
.il { color: #666666 } /* Literal.Number.Integer.Long */

All pau. Now we should have pretty syntax highlighted code! (For those keeping track, this is now version 0.1.3 of my blog.)

Notes on users and groups on Linux

Here are some notes on some basic tasks dealing with users, groups, and permissions on Ubuntu Linux. All these commands (except passwd) are done as root. If you are not root, prepend sudo to all the commands.

  • Add a user, sofeng
    # adduser sofeng
  • Change your password:
    $ passwd
  • Create a developer group
    # addgroup developer
  • Add user sofeng to the developer group
    # adduser sofeng developer
  • Give developer group sudo power:
    Add the following line to /etc/sudoers:
    %developer ALL=(ALL) ALL
  • Change owner to sofeng, and group to developer, of directory, mydir, and all its subdirectories:
    # chown -R sofeng:developer mydir
  • Change permisions of directory, mydir, and all its subdirectories, to be writable by the group, developer:
    # chmod -R g+w mydir

Python recursion example to navigate tree data

Here is a simple Python example using recursion to navigate a nested Python data structure. Each node in the data structure contains 0 or more children. In this simple example, I look at each node and print the "text" indented according to the nesting level within the data structure.

Update 2008-09-15: Nihiliad posted an improvement to my example in the comments. It is much simpler. I have updated my example below.

Nihiliad's (improved) method
data = {'count': 2,
        'text': '1',
        'kids': [{'count': 3,
                  'text': '1.1',
                  'kids': [{'count': 1,
                            'text': '1.1.1',
                            'kids': [{'count':0,
                                      'text': '1.1.1.1',
                                      'kids': []}]},
                           {'count': 0,
                            'text': '1.1.2',
                            'kids': []},
                           {'count': 0,
                            'text': '1.1.3',
                            'kids': []}]},
                 {'count': 0,
                  'text': '1.2',
                  'kids': []}]}

def traverse(data):
    print ' ' * traverse.level + data['text']
    for kid in data['kids']:
        traverse.level += 1
        traverse(kid)
        traverse.level -= 1

if __name__ == '__main__':
    traverse.level = 1
    traverse(data)

Results:

 1
  1.1
   1.1.1
    1.1.1.1
   1.1.2
   1.1.3
  1.2
My original (inferior) method
def outer(data):
    class Namespace: pass
    ns = Namespace()
    ns.level = 1
    
    def inner(data):
        print ' ' * ns.level + data['text']
        if data['count'] > 0:
            ns.level += 1
            for kid in data['kids']:
                inner(kid)
            ns.level -= 1

    inner(data)

if __name__ == '__main__':
    outer(data)

On using Python, the Digg API, and simplejson

Here are some quick notes on using the Digg API with a Python script. Note, there is a Python toolkit for Digg but I just used urllib2 and the Digg API endpoints for the sake of simplicity.

I wanted the output in JSON format so I specified the response type as JSON. To decode JSON directly to a Python data structure, I used simplejson.

Here is a simple example which returns the JSON output for the Digg story Dell vs. Apple: This Time it's Personal which has a "clean title" of Dell_vs_Apple_This_Time_it_s_Personal.

#!/usr/bin/env python

import urllib2

APPKEY = 'http%3A%2F%2Fwww.example.com'
story_clean_title = 'Dell_vs_Apple_This_Time_it_s_Personal'
url = ''.join([
        'http://services.digg.com',
        '/story/%s' % story_clean_title,
        '?appkey=%s' % APPKEY,
        '&type;=json',
        ])
json = urllib2.urlopen(url).read()
print json

Results:

{"timestamp":1219168025,"total":"1","offset":0,"stories":[{"id":"8038250","link":"http:\/\/www.businessweek.com\/magazine\/content\/08_34\/b4097022701166.htm?campaign_id=rss_daily","submit_date":1219047878,"diggs":763,"comments":198,"title":"Dell vs. Apple: This Time it's Personal","description":"Now Bucher is again squaring off against his former company. He's spearheading an ambitious plan at Dell (DELL) to break Apple's dominant hold on the digital entertainment market.","promote_date":1219095692,"status":"popular","media":"news","user":{"name":"msaleem","icon":"http:\/\/digg.com\/users\/msaleem\/l.png","registered":1126518985,"profileviews":136052,"fullname":"Muhammad Saleem"},"topic":{"name":"Apple","short_name":"apple"},"container":{"name":"Technology","short_name":"technology"},"thumbnail":{"originalwidth":370,"originalheight":245,"contentType":"image\/jpeg","src":"http:\/\/digg.com\/apple\/Dell_vs_Apple_This_Time_it_s_Personal\/t.jpg","width":80,"height":80},"href":"http:\/\/digg.com\/apple\/Dell_vs_Apple_This_Time_it_s_Personal"}],"count":1}

Here is a slightly less simple example which returns the comments for the same story above. It uses simplejson to decode the Digg story JSON data and get the story ID which is then used to get the comment data.

#!/usr/bin/env python

import simplejson
import urllib2
from pprint import pprint

APPKEY = 'http%3A%2F%2Fwww.example.com'

def main():
    story_clean_title = 'Dell_vs_Apple_This_Time_it_s_Personal'

    # get story
    json = get_json('/story/%s' % story_clean_title)
    pydata = simplejson.loads(json)
    story_id = pydata['stories'][0]['id']

    # get comments
    json = get_json('/story/%s/comments' % story_id)
    pydata = simplejson.loads(json)
    pprint(pydata)

def get_json(endpoint):
    """ returns json data for requested digg endpoint 
    """
    url = ''.join([
            'http://services.digg.com',
            endpoint,
            '?appkey=%s' % APPKEY,
            '&type;=json',
            ])
    return urllib2.urlopen(url).read()

if __name__ == '__main__':
    main()

Results:

{u'comments': [{u'content': u"For those who are having trouble understanding what this is about.  This is not about creating a competing closed platform.  This is about creating a standard platform for selling music online.  Imagine iTunes but with the ability to add other music stores into it.  So when you want to buy a song, you can browse this music store or that music store.  And all will work with your mp3 player.  Whether it is a basic one that mounts as an external usb drive, or one that is able to sync the songs up the way iTunes does it with the iPod.  And we know it is gonna be good because the guy that is driving it worked at Apple, so he knows what quality is.  I'm sure it will have the ability to import or export your music library to iTunes if you so choose.  But the point is that it is trying to create a standard that any device maker can follow.  And hopefully, it will have no drm.  Otherwise it's as useless as iTunes with its drm.\n\nRemember the same thing happened when windows 3.1 came along and immediately sold as many as 10x the number of pc's compared to mac's at the time.  Anytime there is an open standard, no matter how good the closed standard is, the open one wins because that means lower prices for consumers since any manufacturer can use it.  Apple thrives on closed standards when it comes to selling their products because it enables them to lock their customers in.  And before they know it, they're locked into it and can't get out of it without great expense.\n\nI know that the Apple fanboi's are gonna bury me for this, but I'll say it anyways.  It was nice being on top.  But you can't be on top forever.",
                u'date': 1219163527,
                u'down': 0,
                u'id': 17963749,
                u'level': 0,
                u'replies': 0,
                u'replyto': None,
                u'root': 17963749,
                u'story': 8038250,
                u'up': 1,
                u'user': u'pyrates'},
               {u'content': u'@thinkdifferent: I should of specified its a $1 cheaper when you buy the full album. Single tracks are the same price but albums are usually about $8.99. ',
                u'date': 1219159148,
                u'down': 0,
                u'id': 17961525,
                u'level': 0,
                u'replies': 0,
                u'replyto': None,
                u'root': 17961525,
                u'story': 8038250,
                u'up': 1,
                u'user': u'mrgermy'},
               {u'content': u'Zune XPS w/ Vista Ultimate combo deal.',
                u'date': 1219146354,
                u'down': 0,
                u'id': 17956943,
                u'level': 0,
                u'replies': 0,
                u'replyto': None,
                u'root': 17956943,
                u'story': 8038250,
                u'up': 1,
                u'user': u'hurdboy'},
               {u'content': u'Digg: "Dell vs. Apple: This Time it\'s Personal"\nBusiness Week: "Bucher says his quest to challenge Apple is all business and not personal."\n\nBurried as inacurate.',
                u'date': 1219134278,
                u'down': 0,
                u'id': 17954900,
                u'level': 0,
                u'replies': 0,
                u'replyto': None,
                u'root': 17954900,
                u'story': 8038250,
                u'up': 1,
                u'user': u'KAMiKAZOW'},
               {u'content': u'Dell is junk and Apple is overpriced',
                u'date': 1219117388,
                u'down': 0,
                u'id': 17950148,
                u'level': 0,
                u'replies': 0,
                u'replyto': None,
                u'root': 17950148,
                u'story': 8038250,
                u'up': 1,
                u'user': u'DeuceDiggalow'},
               {u'content': u'Dell, all you have to do to kick apple in the jewels is get your act together with Ubuntu.\n\nAmarok will do the rest.',
                u'date': 1219112434,
                u'down': 2,
                u'id': 17948056,
                u'level': 0,
                u'replies': 4,
                u'replyto': None,
                u'root': 17948056,
                u'story': 8038250,
                u'up': 3,
                u'user': u'ethana2'},
               {u'content': u'Apple = Overpriced, under-featured, but pretty\nDell = priced right, full-featured, and awesome looking',
                u'date': 1219105961,
                u'down': 8,
                u'id': 17945330,
                u'level': 0,
                u'replies': 2,
                u'replyto': None,
                u'root': 17945330,
                u'story': 8038250,
                u'up': 3,
                u'user': u'freesf'},
               {u'content': u'Go Dell! Best of luck to you :)',
                u'date': 1219104145,
                u'down': 0,
                u'id': 17944493,
                u'level': 0,
                u'replies': 0,
                u'replyto': None,
                u'root': 17944493,
                u'story': 8038250,
                u'up': 0,
                u'user': u'Sabre24q7'},
               {u'content': u'Actually the Dell DJ wasnt that bad for its time. Obviously now its looks very dated and far from the best now, but at its time it was a decent player and had pretty good sound quality. It also had more features than the ipod does even today.\n\n Creative was the people who made it and it was a very basic player but it did what it was suppose to and had a lot of room on it for a cheap price. ',
                u'date': 1219104085,
                u'down': 2,
                u'id': 17944474,
                u'level': 0,
                u'replies': 1,
                u'replyto': None,
                u'root': 17944474,
                u'story': 8038250,
                u'up': 1,
                u'user': u'jsc315'},
               {u'content': u"I'm not saying this will or won't succeed. I'm just pointing out something people seem to be missing.\n\nJust because Dell customers aren't pompous and loudmouthed doesn't mean they aren't loyal and happy customers. There are a LOT of Dell users that like the company and what the products they make.",
                u'date': 1219102724,
                u'down': 1,
                u'id': 17943861,
                u'level': 0,
                u'replies': 1,
                u'replyto': None,
                u'root': 17943861,
                u'story': 8038250,
                u'up': 2,
                u'user': u'Urkel'}],
 u'count': 10,
 u'offset': 0,
 u'timestamp': 1219168299,
 u'total': u'55'}

http://www.saltycrane.com is my new OpenID

Open ID is a new technology that allows you to use one set of login credentials to access many sites. This is good because you don't have to remember yet another password or go through yet another Web 2.0 community registration process.

Simon Willison wrote a simple, clear explanation on how to turn your blog in to an OpenID. I followed his instructions, and now http://www.saltycrane.com/ is my new Open ID! I look forward to putting it to good use. (In case you're wondering, yes, I am planning to add Open ID support to this site. Mr. Willison has also written a django-openid library to "enable your Django application to act as an OpenID consumer".)

Somewhere on your Python path

As I install new python packages, I sometimes see instructions which say something like "check out the code, and place it somewhere on your Python path". These are very simple instructions, but since it is not automatic like a Windows installer, or Ubuntu's package management system, it causes me to pause. Where on my Python path should I put it? I could put all my packages in random places and update my PYTHONPATH environment variable every time. I also thought about putting new packages in Python's site-packages directory. This is probably a good option. However, I tend to like to have all my important stuff in my home directory so I can easier maintain it across multiple machines. (Also, I forget where the site-packages lives (it is /usr/lib/python2.5/site-packages on Ubuntu Hardy).) So my solution was to create my own python-packages directory in ~/lib. I set the PYTHONPATH in my ~/.bashrc as follows:

export PYTHONPATH=$HOME/lib/python-packages
and then put all my Python packages here.



Update: Some Python packages are distributed with other stuff besides the actual Python package. To handle these cases, I created a dist directory inside my python-packages directory, and created symbolic links from the actual package directory in dist to python-packages. Finally, I made the entire python-packages directory a Mercurial repository so I can finely control my Python environment and easily maintain it across multiple machines. Here's what my ~/lib/python-packages currently looks like:

drwxr-xr-x 9 sofeng sofeng 4096 2008 08/05 21:40 dist
drwxr-xr-x 4 sofeng sofeng 4096 2008 08/05 21:31 django_openidconsumer
lrwxrwxrwx 1 sofeng sofeng   43 2008 08/05 21:31 elementtree -> dist/elementtree-1.2.6-20050316/elementtree
lrwxrwxrwx 1 sofeng sofeng   31 2008 08/05 21:31 openid -> dist/python-openid-1.2.0/openid
lrwxrwxrwx 1 sofeng sofeng   31 2008 08/05 21:31 openid2.2 -> dist/python-openid-2.2.1/openid
lrwxrwxrwx 1 sofeng sofeng   27 2008 08/05 21:41 pygments -> dist/Pygments-0.10/pygments
lrwxrwxrwx 1 sofeng sofeng   29 2008 08/05 21:31 urljr -> dist/python-urljr-1.0.1/urljr
lrwxrwxrwx 1 sofeng sofeng   29 2008 08/05 21:31 yadis -> dist/python-yadis-1.1.0/yadis

Update 2008-09-14: Here is a post on the django-developers mailing list by Kevin Teague which explains the large number of technologies related to Python package management and deployment including PyPi, Distutils, Eggs, Easy Install, VirtualEnv, and Buildout. Kevin admits that package management and deployment is an area in Python where there is room for a great deal of improvemnt. He notes that the symlinking method that I use can work for simple needs, but it fails for more complicated use cases, such as tracking package dependencies. The new Virtualenv and Buildout technologies seem to be interesting-- I will have to check them out when I have time. I found this link via Simon Willison

Update 2008-09-24: Ian Bicking, author of Virtualenv, has just released pyinstall which seems to be an improved easy_install. I have not tried it yet, but I believe Ian Bicking writes good code.

Update 2008-10-24:Glyph Lefkowitz, lead architect of Twisted, suggests using twisted.python.modules for solving Python Path Programming Problems. This looks like something I could use in my current project-- I just wish I understood it.

Update 2008-12-16: