Databases
StackOverflow2013
Postsdbo
POHow can I prevent excessive RPC calls by pre-fetching ReferenceProperty properties?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow can I prevent excessive RPC calls by pre-fetching ReferenceProperty properties?
    primarykey
    data
    text
    <p>I'm trying to show a table of ~800 entities, and having problems keeping it from being really slow. (Like 15-20 seconds slow.) I successfully implemented memcache, but because I reference a parent model for each of the child entities it still causes a datastore_v3.Get for each of the 800 and is massively slow.</p> <p>I then implemented Nick Johnson's <a href="http://blog.notdot.net/2010/01/ReferenceProperty-prefetching-in-App-Engine" rel="nofollow">ReferenceProperty prefetching</a> and can't solve the following error:</p> <pre><code>[... snipped ...] File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2/webapp2.py", line 570, in dispatch return method(*args, **kwargs) File "/myurl/mypythoncode.py", line 67, in get prefetch_refprops(entitylist, ChildModel.parent_program.name) File "/myurl/mypythoncode.py", line 36, in prefetch_refprops fields = [(entity, prop) for entity in entities for prop in props] TypeError: 'NoneType' object is not iterable </code></pre> <p><strong>Models:</strong></p> <p>These are the two relevant models:</p> <pre><code>class ParentModel(db.Model): name = db.StringProperty() # currently 109 of these class ChildModel(db.Model): name = db.StringProperty() parent_program = db.ReferenceProperty(ParentModel) website = db.StringProperty() # currently 758 of these </code></pre> <p><strong>Python code:</strong></p> <p>In my Python code I'm using Nick Johnson's techniques of <a href="http://blog.notdot.net/2009/9/Efficient-model-memcaching" rel="nofollow">efficient model memcaching</a> and for <a href="http://blog.notdot.net/2010/01/ReferenceProperty-prefetching-in-App-Engine" rel="nofollow">ReferenceProperty prefetching</a>. (I've included the ReferenceProperty prefetching below, but not the memcaching code.)</p> <pre><code>class AllEntities(webapp2.RequestHandler): def get(self): entitylist = deserialize_entities(memcache.get("entitylist")) entityref = prefetch_refprops(entitylist, ChildModel.parent_program.name) if not entitylist: entitylist = ChildModel.all().fetch(None) entityref = prefetch_refprops(entitylist, ChildModel.parent_program.name) memcache.set("entitylist", serialize_entities(entitylist)) context = { 'entitylist': entitylist, } self.response.out.write(template.render(context)) def prefetch_refprops(entities, *props): fields = [(entity, prop) for entity in entities for prop in props] ref_keys = [prop.get_value_for_datastore(x) for x, prop in fields] ref_entities = dict((x.key(), x) for x in db.get(set(ref_keys))) for (entity, prop), ref_key in zip(fields, ref_keys): prop.__set__(entity, ref_entities[ref_key]) return entities </code></pre> <p><strong>Jinja2 template:</strong></p> <p>My Jinja2 template references the iterable "entry" in "entitylist" but also the parent_program.name and parent_program.key().id()</p> <pre><code>{% for entry in entitylist %} &lt;tr&gt; &lt;td&gt;&lt;a href="{{ entry.website}}"&gt;{{ entry.website }}&lt;/a&gt;&lt;/td&gt; &lt;td&gt;&lt;a href="/urlcategory/view?entityid={{ entry.parent_program.key().id() }}"&gt;{{ entry.parent_program.name }}&lt;/td&gt; &lt;/tr&gt; {% endfor %} </code></pre> <p>I've replaced the line:</p> <pre><code>entityref = prefetch_refprops(entitylist, ChildModel.parent_program.name) </code></pre> <p>with</p> <pre><code>entityref = prefetch_refprops(entitylist, ChildModel.parent_program) </code></pre> <p>and other variations that include ".name" and ".key().id()". When I use ".key().id()" I get the error:</p> <pre><code>AttributeError: 'ReferenceProperty' object has no attribute 'key' </code></pre> <p>What am I missing or screwing up? I'd really appreciate any help!</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJed Christiansen
    1. USJed Christiansen
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. CODelete the first `entityref = prefetch_refprops(entitylist, ChildModel.parent_program.name)` line. `entitylist` is set to None the first time through, so the prefetch fails.
      singulars
      1. POHow can I prevent excessive RPC calls by pre-fetching ReferenceProperty properties?
      1. USmjibson
    2. COThanks, I've fixed that. But now I'm getting "AttributeError: 'str' object has no attribute 'get_value_for_datastore'" for "prefetch_refprops(companylist, ChildModel.parent_program.name)". This seems odd since it's virtually the exact same as the same that Nick uses in his example.
      singulars
      1. POHow can I prevent excessive RPC calls by pre-fetching ReferenceProperty properties?
      1. USJed Christiansen
    3. COOkay, I fixed it now, by changing it to "prefetch_refprops(companylist, ChildModel.parent_program)". I'm not getting any more errors, but it feels like I'm still not doing something right...
      singulars
      1. POHow can I prevent excessive RPC calls by pre-fetching ReferenceProperty properties?
      1. USJed Christiansen
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload