Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>If your data doesn't need to be updated more than once per-minute per user:</p> <ol> <li>Setup the "alerts" in a <code>LocalStructuredProperty</code> on the user.</li> <li><p>When "putting" an incoming data point from the feed, use a pre-put hook to pre-calculate the values:</p> <ul> <li>Grab the user entity in the pre-put hook. (If using NDB and you already grab the user, it should come from local memory)</li> <li>Grab all of the "alerts" for that user and process them asynchronously (tasklets)</li> <li>Store everyone's alert data in its own entity, use special key-names to make querying fast (e.g. set they keyname something like <code>&lt;user&gt;_&lt;alert_type&gt;_&lt;time_in_seconds&gt;_&lt;percentage&gt;</code> so you can do a <code>get</code> instead of a <code>query</code>. In this object, store all the data points that come in and fall within the time-limit specified. For one update every minute, you can probably store 1000+ datapoints as a list of tuples <code>(&lt;timestamp&gt;, &lt;value&gt;)</code>. From this process the alert based on the configuration defined and store the new value.</li> </ul></li> </ol> <p>Example (tbh. this is a rough example. Should use transactions if you want guarantees on the data):</p> <pre><code>class AlertConfiguration(ndb.Model): timespan_in_seconds = ndb.IntegerProperty('tis', indexed=False) percent_change = ndb.FloatProperty('pc', indexed=False) class User(ndb.Model): alerts = LocalStructuredProperty(AlertConfiguration, repeated=True, name='a') ... class DataPoint(ndb.Model): timestamp = ndb.DateTimeProperty('ts', auto_now_add=True) value = ndb.FloatProperty('v') user = ndb.KeyProperty(name='u', kind=User) def _pre_put_hook(self): alerts = self.user.get().alerts futures = [] for alert in alerts: futures.append(process_alert(alert, self)) yield futures class AlertProcessor(ndb.Model): previous_data_points = ndb.JsonProperty(name='pdp', compressed=True) @ndb.tasklet def process_alert(alert_config, data_point): key_name = '{user}_{timespan}_{percentage}'.format(user=data_point.user.id(), timespan=alert_config.timespan_in_seconds, percentage=alert_config.percent_change) processor = yield AlertProcessor.get_or_insert_async(key_name) new_points = [] found = False for point in processor.previous_data_points: delta = data_point.timestamp - datetime.strptime(point[0], '%c') seconds_diff = (86400 * delta.days) + delta.seconds if seconds_diff &lt; alert_config.timespan_in_seconds: new_points.add(point) if not found: found = True if (data_point.value - point[1]) / data_point.value &gt;= alert_config.percent_change: #E-mail alert here? new_points.append((data_point.timestamp.strftime('%c'), data_point.value)) processor.previous_data_points = new_points yield processor.put_async() </code></pre>
    singulars
    1. This table or related slice is empty.
    1. POQuerying for Percent of Change Over Time in App Engine
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USsomeone1
    1. USsomeone1
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. COThanks @someone1, Three questions: First, why make the data points per user? the system has one global data feed. Second, I'm worried that this won't scale well for thousands of users. Finally, I don't see how is this better then just querying the one feed, per alert on a new datapoint put.
      singulars
      1. PO
      1. USyoav.aviram
    2. CO1. Well following your Twitter example, I assumed the data was coming in on a per-user basis. If its global, then its global... My only concern then would be how do you know which users have alerts setup on the global feed, do the data-points tell you? I'd probably define a set of alerts to subscribe to and limit my users to that manageable set. 2. This tries to minimize RPC time and put most of the burden on memory/CPU. 3. Well a query will count as 1 read for the query + 1 read for every returned object. Queries also take longer than `get` calls.
      singulars
      1. PO
      1. USsomeone1
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload