Databases
StackOverflow2013
Postsdbo
POSubclassing db.TextProperty for storing python dict as JSON and setting default encoding to anything but ASCII
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSubclassing db.TextProperty for storing python dict as JSON and setting default encoding to anything but ASCII
    primarykey
    data
    text
    <p>Using Google App Engine (python SDK), I created a custom JSONProperty() as a subclass of db.TextProperty(). My goal is to store a python dict on the fly as JSON and retrieve it easily. I followed various examples found via Google and setting up the custom Property class and methods is pretty easy.</p> <p>However, some of my dict values (strings) are encoded in utf-8. When saving the model into the datastore, I get a dreaded Unicode error (for datastore text property default encoding is ASCII). Subclassing db.BlobProperty didn't solve the issue.</p> <p>Basically, my code does the following thing : store Resource entities into the datastore (with URL as a StringProperty and POST/GET payloads stored in a dict as a JSONProperty), fetch them later (code not included). I choose not to use pickle for storing payloads because I'm a JSON freak and have no use storing objects.</p> <p>Custom JSONProperty :</p> <pre><code>class JSONProperty(db.TextProperty): def get_value_for_datastore(self, model_instance): value = super(JSONProperty, self).get_value_for_datastore(model_instance) return json.dumps(value) def make_value_from_datastore(self, value): if value is None: return {} if isinstance(value, basestring): return json.loads(value) return value </code></pre> <p>Putting model into datastore :</p> <pre><code>res = Resource() res.init_payloads() res.url = "http://www.somesite.com/someform/" res.param = { 'name': "SomeField", 'default': u"éàôfoobarç" } res.put() </code></pre> <p>This will throw a UnicodeDecodeError related to ASCII encoding. Maybe it's worth noting that I only get this error (everytime) on production server. I'm using python 2.5.2 on dev.</p> <blockquote> <p>Traceback (most recent call last): File "/base/data/home/apps/delpythian/1.350065314722833389/core/handlers/ResetHandler.py", line 68, in _res_one return res_one.put() File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/<strong>init</strong>.py", line 984, in put return datastore.Put(self._entity, config=config) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 455, in Put return _GetConnection().async_put(config, entities, extra_hook).get_result() File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1219, in async_put for pbs in pbsgen: File "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1070, in __generate_pb_lists pb = value_to_pb(value) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 239, in entity_to_pb return entity._ToPb() File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore.py", line 841, in _ToPb properties = datastore_types.ToPropertyPb(name, values) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore_types.py", line 1672, in ToPropertyPb pbvalue = pack_prop(name, v, pb.mutable_value()) File "/base/python_runtime/python_lib/versions/1/google/appengine/api/datastore_types.py", line 1485, in PackString pbvalue.set_stringvalue(unicode(value).encode('utf-8')) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32: ordinal not in range(128)</p> </blockquote> <p>My question is the following : is there a way to subclass a db.TextProperty() class and set/enforce a custom encoding ? Or am I doing something wrong ? I try to avoid using str() and follow the <em>"Decode early, Unicode everywhere, encode late"</em> rule.</p> <p>Update : added code and stacktrace.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USjbmusso
    1. USjbmusso
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSubclassing db.TextProperty for storing python dict as JSON and setting default encoding to anything but ASCII
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POSubclassing db.TextProperty for storing python dict as JSON and setting default encoding to anything but ASCII
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POSubclassing db.TextProperty for storing python dict as JSON and setting default encoding to anything but ASCII
      1. USFederico Elles
      1. VTFavorite
    1. COPlease add code samples and a stack trace. In particular, I'd like to confirm that the UnicodeError happens when you try to store the TextProperty and not earlier when you try to serialize your Python dictionary to JSON.
      singulars
      1. POSubclassing db.TextProperty for storing python dict as JSON and setting default encoding to anything but ASCII
      1. USDrew Sears
    2. COUpdated question, added code and stack trace.
      singulars
      1. POSubclassing db.TextProperty for storing python dict as JSON and setting default encoding to anything but ASCII
      1. USjbmusso
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload