SaltyCrane Blog — Notes on JavaScript and web development

Saving a Python dict to a file using pickle

Date:  |  Modified:  |  Tags: persistence, python  |  16 Comments

Per Programming Python, 3rd Edition, there are a number of methods to store persistent data with Python:

  • I often use flat files to read or write text (string) data using the os library.
  • Flat files are read sequentially, but dbm files allow for keyed access to string data
  • The pickle module can be used to store non-string Python data structures, such as Python dicts. However, the data is not keyed as with dbm files.
  • shelve files combine the best of the dbm and pickle methods by storing pickled objects in dbm keyed files.
  • I've read good things about the ZODB object-oriented database, but I don't know too much about it. Per the book, it is a more powerful alternative to shelves.
  • The final option is interfacing with a full-fledged SQL relational databases. As I mentioned before, Python 2.5 has an interface to SQLite as part of the standard distribution.

Here is an example using pickle which writes a Python dict to a file and reads it back again:

import pickle

# write python dict to a file
mydict = {'a': 1, 'b': 2, 'c': 3}
output = open('myfile.pkl', 'wb')
pickle.dump(mydict, output)
output.close()

# read python dict back from the file
pkl_file = open('myfile.pkl', 'rb')
mydict2 = pickle.load(pkl_file)
pkl_file.close()

print mydict
print mydict2

Results:
{'a': 1, 'c': 3, 'b': 2}
{'a': 1, 'c': 3, 'b': 2}

Comments


#1 code43 commented on :

hi Eliot, here's an appropriate Python module for you:

http://yserial.sourceforge.net

The module is instructive in the way it unifies the standard batteries: sqlite3 (as of Python v2.5), zlib (for compression), and cPickle (for securely serializing objects).

If your Python program requires data persistance, then y_serial is a module which should be worth importing. Objects are warehoused in a database file in very compressed form. Steps for insertion, organization by annotation, and finally retrieval are amazingly simple.

Hope this helps...


#2 Eliot commented on :

code43, looks good, thanks!


#3 __django__ commented on :

Thanks for code. I looking for.


#4 PATX commented on :

I always refer back to this post every time I do something with dicts... Well most of the time :P. I'm forgetful and this is super helpful.


#5 james commented on :

Hello ,

How to append dictionary values to a file?

I mean how to add dictionary values to a file without overwritting the existing dictionary values


#6 Chrisfs commented on :

James, If you want to append data to a text file, you want to open it with the option 'a'. So the code would look like newfile = open('myfile.txt',a)

Then when you wrote the data back, it would append it. I don't know how well that would work with a pickled file.


#7 Eliot commented on :

James:

Maybe you want something like this?

import pickle

# write python dict to a file
mydict = {'a': 1, 'b': 2, 'c': 3}
output = open('myfile.pkl', 'wb')
pickle.dump(mydict, output)
output.close()
print mydict

# read python dict back from the file
pkl_file = open('myfile.pkl', 'rb')
mydict = pickle.load(pkl_file)
pkl_file.close()
print mydict

# update dict and write to the file again
mydict.update({'d': 4})
output = open('myfile.pkl', 'wb')
pickle.dump(mydict, output)
output.close()

# read python dict back from the file
pkl_file = open('myfile.pkl', 'rb')
mydict = pickle.load(pkl_file)
pkl_file.close()
print mydict

Results:

{'a': 1, 'c': 3, 'b': 2}
{'a': 1, 'c': 3, 'b': 2}
{'a': 1, 'c': 3, 'b': 2, 'd': 4}

#8 Jose commented on :

How to load dictionaries from a pickled file that contains many dictionaries. So far I have been able to load only the first dictionary, and I don't know how to iterate file to load every dictionary and print it.Please help :)


#9 raja commented on :

Hii,what if wants to replace the value of a particular key of the dictionary,i messed with this.


#10 gudGuy commented on :

@Jose

well may be i don't know the exact method to save and retrieve multiple dictionaries in pickle, but one thing you can do is that you can make dictionary of dictionaries.

e.g d = {'one': {'a': 1, 'b': 2}, 'two': {'c': 3, 'e': 4}}


#11 Paul commented on :

@Jose

You can just use pickle.load repeatedly until you get an EOFError exception. Each successive call will give you the next pickled object in the file.


#12 Charl commented on :

Thanks guy, this works great for loading dictionaries as fixtures for unit tests.


#13 Eliot commented on :

Charl, I'm glad that worked good for you. Another option for saving fixture data for unit tests is saving the data as json in a file with e.g. the json module in the standard library: http://docs.python.org/library/json.html (I know that method wasn't in the list above. :)


#14 Nitrogen commented on :

Dear Elliot, Thanks a lot for this post. I have been working on this for days and still not able to figure it out. However I have a similar Challenge to what Jose has posted above. I am trying to save these details(id, name, amount)about each customer in the flat file and then read a particular record if say it's id is typed in. so for example i receive the customer id as a raw_input and and display his name and amount.


#15 Nathaniel commented on :

can you type a value e.g 1,2,3,and display it under read using pickle?
displaying the value in numbers and not in binary?

disqus:1958257677


#16 Daddy commented on :

Writing a post to show how to pickle? Really?

disqus:3704443135