Databases
StackOverflow2013
Postsdbo
POWhat response times can be expected from GAE/NDB?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhat response times can be expected from GAE/NDB?
    primarykey
    data
    text
    <p>We are currently building a small and simple central HTTP service that maps "external identities" (like a facebook id) to an "internal (uu)id", unique across all our services to help with analytics.</p> <p>The first prototype in "our stack" (flask+postgresql) was done within a day. But since we want the service to (almost) never fail and scale automagically, we decided to use Google App Engine.</p> <p>After a week of reading&amp;trying&amp;benchmarking this question emerges:</p> <p><strong>What response times are considered "normal" on App Engine (with NDB)?</strong></p> <p>We are getting response times that are consistently <em>above 500ms on average</em> and <em>well above 1s in the 90percentile</em>.</p> <p>I've attached a stripped down version of our code below, hoping somebody can point out the obvious flaw. We really like the autoscaling and the distributed storage, but we can not imagine 500ms really is the expected performance in our case. The sql based prototype responded much faster (consistently), hosted on one single Heroku dyno using the free, cache-less postgresql (even with an ORM).</p> <p>We tried both synchronous and asynchronous variants of the code below and looked at the appstats profile. It's always RPC calls (both memcache and datastore) that take very long (50ms-100ms), made worse by the fact that there are always multiple calls (eg. mc.get() + ds.get() + ds.set() on a write). We also tried deferring as much as possible to the task queue, without noticeable gains.</p> <pre class="lang-py prettyprint-override"><code>import json import uuid from google.appengine.ext import ndb import webapp2 from webapp2_extras.routes import RedirectRoute def _parse_request(request): if request.content_type == 'application/json': try: body_json = json.loads(request.body) provider_name = body_json.get('provider_name', None) provider_user_id = body_json.get('provider_user_id', None) except ValueError: return webapp2.abort(400, detail='invalid json') else: provider_name = request.params.get('provider_name', None) provider_user_id = request.params.get('provider_user_id', None) return provider_name, provider_user_id class Provider(ndb.Model): name = ndb.StringProperty(required=True) class Identity(ndb.Model): user = ndb.KeyProperty(kind='GlobalUser') class GlobalUser(ndb.Model): uuid = ndb.StringProperty(required=True) @property def identities(self): return Identity.query(Identity.user==self.key).fetch() class ResolveHandler(webapp2.RequestHandler): @ndb.toplevel def post(self): provider_name, provider_user_id = _parse_request(self.request) if not provider_name or not provider_user_id: return self.abort(400, detail='missing provider_name and/or provider_user_id') identity = ndb.Key(Provider, provider_name, Identity, provider_user_id).get() if identity: user_uuid = identity.user.id() else: user_uuid = uuid.uuid4().hex GlobalUser( id=user_uuid, uuid=user_uuid ).put_async() Identity( parent=ndb.Key(Provider, provider_name), id=provider_user_id, user=ndb.Key(GlobalUser, user_uuid) ).put_async() return webapp2.Response( status='200 OK', content_type='application/json', body = json.dumps({ 'provider_name' : provider_name, 'provider_user_id' : provider_user_id, 'uuid' : user_uuid }) ) app = webapp2.WSGIApplication([ RedirectRoute('/v1/resolve', ResolveHandler, 'resolve', strict_slash=True) ], debug=False) </code></pre> <p>For completeness sake the (almost default) app.yaml</p> <pre><code>application: GAE_APP_IDENTIFIER version: 1 runtime: python27 api_version: 1 threadsafe: yes handlers: - url: .* script: main.app libraries: - name: webapp2 version: 2.5.2 - name: webob version: 1.2.3 inbound_services: - warmup </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. UStinnet
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POWhat response times can be expected from GAE/NDB?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POWhat response times can be expected from GAE/NDB?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POWhat response times can be expected from GAE/NDB?
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload