Databases
StackOverflow2013
Postsdbo
POAkka for REST polling
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POAkka for REST polling
    primarykey
    data
    text
    <p>I'm trying to interface a large Scala + Akka + PlayMini application with an external REST API. The idea is to periodically poll (basically every 1 to 10 minutes) a root URL and then crawl through sub-level URLs to extract data which is then sent to a message queue.</p> <p>I have come up with two ways to do this:</p> <h2>1st way</h2> <p>Create a hierarchy of actors to match the resource path structure of the API. In the Google Latitude case, that would mean, e.g.</p> <ul> <li>Actor 'latitude/v1/currentLocation' polls <a href="https://www.googleapis.com/latitude/v1/currentLocation" rel="noreferrer">https://www.googleapis.com/latitude/v1/currentLocation</a></li> <li>Actor 'latitude/v1/location' polls <a href="https://www.googleapis.com/latitude/v1/location" rel="noreferrer">https://www.googleapis.com/latitude/v1/location</a></li> <li>Actor 'latitude/v1/location/1' polls <a href="https://www.googleapis.com/latitude/v1/location/1" rel="noreferrer">https://www.googleapis.com/latitude/v1/location/1</a></li> <li>Actor 'latitude/v1/location/2' polls <a href="https://www.googleapis.com/latitude/v1/location/2" rel="noreferrer">https://www.googleapis.com/latitude/v1/location/2</a></li> <li>Actor 'latitude/v1/location/3' polls <a href="https://www.googleapis.com/latitude/v1/location/3" rel="noreferrer">https://www.googleapis.com/latitude/v1/location/3</a></li> <li>etc.</li> </ul> <p>In this case, each actor is responsible for polling its associated resource periodically, as well as creating / deleting child actors for next-level path resources (i.e. actor 'latitude/v1/location' creates actors 1, 2, 3, etc. for all locations it learns about through polling of <a href="https://www.googleapis.com/latitude/v1/location" rel="noreferrer">https://www.googleapis.com/latitude/v1/location</a>).</p> <h2>2nd way</h2> <p>Create a pool of identical polling actors which receive polling requests (containing the resource path) load-balanced by a router, poll the URL once, do some processing, and schedule polling requests (both for next-level resources and for the polled URL). In Google Latitude, that would mean for instance:</p> <p>1 router, n poller actors. Initial polling request for <a href="https://www.googleapis.com/latitude/v1/location" rel="noreferrer">https://www.googleapis.com/latitude/v1/location</a> leads to several new (immediate) polling requests for <a href="https://www.googleapis.com/latitude/v1/location/1" rel="noreferrer">https://www.googleapis.com/latitude/v1/location/1</a>, <a href="https://www.googleapis.com/latitude/v1/location/2" rel="noreferrer">https://www.googleapis.com/latitude/v1/location/2</a>, etc. and one (delayed) polling request for the same resource, i.e. <a href="https://www.googleapis.com/latitude/v1/location" rel="noreferrer">https://www.googleapis.com/latitude/v1/location</a>.</p> <p>I have implemented both solutions and can't immediately observe any relevant difference of performance, at least not for the API and polling frequencies I am interested in. I find the first approach to be somewhat easier to reason about and perhaps easier to use with system.scheduler.schedule(...) than the second approach (where I need to scheduleOnce(...)). Also, assuming resources are nested through several levels and somewhat short-lived (e.g. several resources may be added/removed between each polling), akka's lifecycle management makes it easy to kill off a whole branch in the 1st case. The second approach should (theoretically) be faster and the code is somewhat easier to write.</p> <p>My questions are: </p> <ol> <li>What approach seems to be the best (in terms of performance, extensibility, code complexity, etc.)?</li> <li>Do you see anything wrong with the design of either approach (esp. the 1st one)?</li> <li>Has anyone tried to implement anything similar? How was it done?</li> </ol> <p>Thanks!</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USTommi
    1. USuser1403269
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POAkka for REST polling
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POAkka for REST polling
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POAkka for REST polling
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI'd be more worried about the error paths. What happens if there's a transient failure of a particular intermediate path; do you retry? Will the failure of any given child actor also cause the failure of the parent actor? At sufficient scale you're almost guaranteed to have _some_ transient failures by children.
      singulars
      1. POAkka for REST polling
      1. USRoss Judson
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload