StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>This example uses <a href="http://spray.io/documentation/spray-client/#message-pipeline" rel="nofollow">spray-client</a>, <a href="http://doc.akka.io/docs/akka/2.1.0/scala/futures.html" rel="nofollow">scala futures</a>, and <a href="http://doc.akka.io/docs/akka/2.1.0/scala/scheduler.html" rel="nofollow">Akka scheduler</a>. </p> <p>Implementation varies depending on desired behavior (execute many requests in parallel at the same time, execute in different intervals, send responses to one actor to process one response at a time, send responses to many actors to process in parallel... etc).</p> <p>This particular example shows how execute many requests in parallel at the same time, and then do something with each result as it completes, without waiting for any other requests that were fired off at the same time to complete.</p> <p>The code below will execute two HTTP requests every 5 seconds to 0.0.0.0:9000/helloWorld and 0.0.0.0:9000/goodbyeWorld in parallel.</p> <p>Tested in Scala 2.10, Spray 1.1-M7, and Akka 2.1.2:</p> <p>Actual scheduling code that handles periodic job execution:</p> <pre><code>// Schedule a periodic task to occur every 5 seconds, starting as soon // as this schedule is registered system.scheduler.schedule(initialDelay = 0 seconds, interval = 5 seconds) { val paths = Seq("helloWorld", "goodbyeWorld") // perform an HTTP request to 0.0.0.0:9000/helloWorld and // 0.0.0.0:9000/goodbyeWorld // in parallel (possibly, depending on available cpu and cores) val retrievedData = Future.traverse(paths) { path => val response = fetch(path) printResponse(response) response } } </code></pre> <p>Helper methods / boilerplate setup:</p> <pre><code>// Helper method to fetch the body of an HTTP endpoint as a string def fetch(path: String): Future[String] = { pipeline(HttpRequest(method = GET, uri = s"/$path")) } // Helper method for printing a future'd string asynchronously def printResponse(response: Future[String]) { // Alternatively, do response.onComplete {...} for (res <- response) { println(res) } } // Spray client boilerplate val ioBridge = IOExtension(system).ioBridge() val httpClient = system.actorOf(Props(new HttpClient(ioBridge))) // Register a "gateway" to a particular host for HTTP requests // (0.0.0.0:9000 in this case) val conduit = system.actorOf( props = Props(new HttpConduit(httpClient, "0.0.0.0", 9000)), name = "http-conduit" ) // Create a simple pipeline to deserialize the request body into a string val pipeline: HttpRequest => Future[String] = { sendReceive(conduit) ~> unmarshal[String] } </code></pre> <p>Some notes:</p> <ul> <li><p><code>Future.traverse</code> is used for running futures in parallel (ignores order). Using a for comprehension on a list of futures will execute one future at a time, waiting for each to complete.</p> <pre><code>// Executes `oneThing`, executes `andThenAnother` when `oneThing` is complete, // then executes `finally` when `andThenAnother` completes. for { oneThing <- future1 andThenAnother <- future2 finally <- future3 } yield (...) </code></pre></li> <li><p><code>system</code> will need to be replaced with your actual Akka actor system.</p></li> <li><p><code>system.scheduler.schedule</code> in this case is executing an arbitrary block of code every 5 seconds -- there is also an overloaded version for scheduling messages to be sent to an actorRef.</p> <pre><code>system.scheduler.schedule( initialDelay = 0 seconds, frequency = 30 minutes, receiver = rssPoller, // an actorRef message = "doit" // the message to send to the actorRef ) </code></pre></li> <li><p>For your particular case, printResponse can be replaced with an actor send instead: <code>anActorRef ! response</code>.</p></li> <li>The code sample doesn't take into account failures -- a good place to handle failures would be in the printResponse (or equivalent) method, by using a Future onComplete callback: <code>response.onComplete {...}</code></li> <li>Perhaps obvious, but spray-client can be replaced with another http client, just replace the <code>fetch</code> method and accompanying spray code.</li> </ul> <p><strong>Update:</strong> Full running code example is <a href="https://github.com/daniel-trinh/webcrawler/blob/840a4bc50430b02ccc777c2637b53c89dc24d54a/app/biz/HttpPoller.scala" rel="nofollow">here</a>:</p> <p>git clone the repo, checkout the specified commit sha, <code>$ sbt run</code>, navigate to <code>0.0.0.0:9000</code>, and watch the code in the console where <code>sbt run</code> was executed -- it should print <code>Hello World!\n'Goodbye World!</code> OR <code>Goodbye World!\nHelloWorld!</code> (order is potentially random because of parallel Future.traverse execution).</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload