Card store project #4: Notes on using Amazon's CloudFront

I haven't been keeping up with the current events very well recently, but I haven't noticed a lot of people using Amazon's S3 or CloudFront with Django on VPS hosting. Though there is Adrian's post from 2006, I see more articles about serving media files with lighttpd or, more recently, nginx. Is a CDN unnecessary for our needs? I thought it'd be good to take some load off my VPS server since I need all the memory I can get for my Django web server and database. But maybe web servers such as nginx are so lightweight it doesn't make much of an impact? I didn't think the cost would be too much-- on this blog, I'm only paying about $0.10/month for S3 services to serve my static media. Of course, there isn't a lot of static media to serve on this blog, but it still seems like it would be a fraction of the $20/month I'm paying for VPS at Slicehost. It may be the convenience factor-- because every time I update a static file, I then have to upload it to S3. This is even more inconvenient for files uploaded through the admin interface. I think some people have probably solved this already... maybe using Django signals. Maybe it is a combination of all these things. Please let me know what you think. If you're not using S3/CloudFront, why aren't you?

Well I went ahead and gave CloudFront a try since it is so easy. My card store project website seems to be somewhat faster than before. Please check it out here. I'm still not sure if I should be happy with the site's speed though. I did a quick memcached install, but I don't think I've configured it properly. I will probably need to revisit that. Anyways, here are my notes on using CloudFront with my Satchmo store.

Sign up for S3

• Sign up for Amazon Web Services
• Sign up for Simple Storage Service
• Take note of your "Access Key ID" and your "Secret Access Key" under "Your Account", "Access Identifiers"

Get S3 Python library

• Download the Amazon S3 Python library
• Unpack it, and put s3-example-libraries/python/S3.py somewhere on your Python path.

Create a S3 bucket

• Create a file named create_bucket.py:

  import S3
  
  ACCESS_KEY = 'myaccesskey'
  SECRET_KEY = 'mysecretaccesskey'
  BUCKET_NAME = 'handsoncards'
  
  conn = S3.AWSAuthConnection(ACCESS_KEY, SECRET_KEY)
  conn.create_bucket(BUCKET_NAME)

• Run it:

  python create_bucket.py

Upload files to S3

• Download Adrian's S3 upload script and save it to /srv/HandsOnCards/handsoncards/bin/update_s3.py
• Edit the script with the correct values for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and BUCKET_NAME.
• Upload files. (Assumes static directory is linked to /var/www/site_media).

  cd /var/www
  find -L site_media | grep -v '~$' | python /srv/HandsOnCards/handsoncards/bin/update_s3.py
  find -L admin_media | grep -v '~$' | python /srv/HandsOnCards/handsoncards/bin/update_s3.py

Set up CloudFront

• Sign up for CloudFront
• Get the S3 Fox firefox plugin
• Click "Manage Accounts" and enter access key and secret key
• Right click on your bucket (handsoncards) and select "Manage Distributions" Enter a "Comment" and optional CNAME, then click "Create Distribution".
• Wait a while while the distribution is created. Take note of the "Domain Name". For me it is: http://d16z1yuk7jeryy.cloudfront.net
• Click the refresh button until the "Status" says "Deployed"

Update settings and templates to use CloudFront

• In settings.py set MEDIA_URL and ADMIN_MEDIA_PREFIX as follows:

  MEDIA_URL = 'http://d16z1yuk7jeryy.cloudfront.net/site_media/'
  ADMIN_MEDIA_PREFIX = 'http://d16z1yuk7jeryy.cloudfront.net/admin_media/'

• In your base.html template and all other templates, replace /site_media/ with http://d16z1yuk7jeryy.cloudfront.net/site_media/.

Update 2009-06-08: Add "Expires" headers

For better performance, it is good to add a far-future "Expires" header to static content on S3. To do this I modified Adrian's script to set the "Expires" header to be one year in the future as shown below. Thanks to orip for this tip.

from datetime import datetime, timedelta
import mimetypes
import os.path
import sys
import S3 # Get this from Amazon

AWS_ACCESS_KEY_ID = 'CHANGEME'
AWS_SECRET_ACCESS_KEY = 'CHANGEME'
BUCKET_NAME = 'CHANGEME'

def update_s3():
    conn = S3.AWSAuthConnection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    for line in sys.stdin:
        filename = os.path.normpath(line[:-1])
        if filename == '.' or not os.path.isfile(filename):
            continue # Skip this, because it's not a file.
        print "Uploading %s" % filename
        filedata = open(filename, 'rb').read()
        expires = datetime.utcnow() + timedelta(days=365)
        expires = expires.strftime("%a, %d %b %Y %H:%M:%S GMT")
        content_type = mimetypes.guess_type(filename)[0]
        if not content_type:
            content_type = 'text/plain'
        conn.put(BUCKET_NAME, filename, S3.S3Object(filedata),
            {'x-amz-acl': 'public-read', 
             'Content-Type': content_type,
             'Expires': expires,
             })

if __name__ == "__main__":
    update_s3()

• "Add an Expires or a Cache-Control Header" section of the Yahoo Developer Network "Best Practices for Speeding Up Your Web Site" guide
• Section 14.21 of the HTTP specification
• Rob Larsen's blog post: "Setting Far Future Expires Headers For Images In Amazon S3"

Update 2009-10-21: Add CNAME record

Go to your DNS Zone manager and add a CNAME record with the following parameters:

• Type: CNAME
• Name: static
• Data: d16z1yuk7jeryy.cloudfront.net. (Don't forget the period at the end!)
• TTL: whatever you want. I left it at 86400

Now wherever I previously would have used http://d16z1yuk7jeryy.cloudfront.net, I can replace it with http://static.handsoncards.com.