GAE: Storing serializable objects in datastore

Google AppEngine’s datastore supports a variety of simple Types and Property Classes by default. However if we want to store something like a dictionary, we typically have to serialize it and store it as a Blob. On fetching we will de-serialize it. While this approach works, it is repetitive and somewhat error-prone.

Wouldn’t it be great if there is a SerializableProperty class that can handle this automatically for us? It doesn’t exist but according to this article, it is easy to create our own customized Property classes. So here is a simple implementation of SerializableProperty that worked for me:

import cPickle as pickle
import zlib
from google.appengine.ext import db

class SerializableProperty(db.Property):
  """
  A SerializableProperty will be pickled and compressed before it is
  saved as a Blob in the datastore. On fetch, it would be decompressed
  and unpickled. It allows us to save serializable objects (e.g. dicts)
  in the datastore.

  The sequence of transformations applied can be customized by calling
  the set_transforms() method.
  """

  data_type = db.Blob
  _tfm = [lambda x: pickle.dumps(x,2), zlib.compress]
  _itfm = [zlib.decompress, pickle.loads]

  def set_transforms(self, tfm, itfm):
    self._tfm = tfm
    self._itfm = itfm

  def get_value_for_datastore(self, model_instance):
    value = super(SerializableProperty,
        self).get_value_for_datastore(model_instance)
    if value is not None:
      value = self.data_type(reduce(lambda x,f: f(x), self._tfm, value))
    return value

  def make_value_from_datastore(self, value):
    if value is not None:
      value = reduce(lambda x,f: f(x), self._itfm, value)
    return value

Usage is as simple as this:

class MyModel(db.Model):
  data = SerializableProperty()

entity = MyModel(data = {"key": "value"}, key_name="somekey")
entity.put()
entity = MyModel.get_by_key_name("somekey")
print entity.data

Hope that helps!

Update (20091126): I’ve changed db.Blob to self.data_type as suggested by Peritus in Comment. The same comment also suggested a JSONSerializableProperty subclass:

import simplejson as json
class JSONSerializableProperty(SerializableProperty):
  data_type = db.Text
  _tfm = [json.dumps]
  _itfm = [json.loads]

Thanks Peritus!