StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>More info on this issue.</p> <p>From Rapgenius: <a href="http://rapgenius.com/Lemon-money-trees-rap-genius-response-to-heroku-lyrics" rel="nofollow">http://rapgenius.com/Lemon-money-trees-rap-genius-response-to-heroku-lyrics</a></p> <blockquote> <p>Ten days ago, spurred by a minor problem serving our compiled javascript, we started running a lot of ab benchmarks. We noticed that the numbers we were getting were consistently worse than the numbers reported to us by Heroku and their analytics partner New Relic. For a static copyright page, for instance, Heroku reported an average response time of 40ms; our tools said 6330ms. What could account for such a big difference?</p> <p>“Requests are waiting in a queue at the dyno level,” a Heroku engineer told us, “then being served quickly (thus the Rails logs appear fast), but the overall time is slower because of the wait in the queue.”</p> <p>Waiting in a queue at the dyno level? What?</p> </blockquote> <p>From Heroku: <a href="https://blog.heroku.com/archives/2013/2/16/routing_performance_update" rel="nofollow">https://blog.heroku.com/archives/2013/2/16/routing_performance_update</a></p> <blockquote> <p>Over the past couple of years Heroku customers have occasionally reported unexplained latency on Heroku. There are many causes of latency—some of them have nothing to do with Heroku—but until this week, we failed to see a common thread among these reports. We now know that our routing and load balancing mechanism on the Bamboo and Cedar stacks created latency issues for our Rails customers, which manifested themselves in several ways, including:</p> <ul> <li>Unexplainable, high latencies for some requests</li> <li>Mismatch between reported queuing and service time metrics and the observed reality</li> <li>Discrepancies between documented and observed behaviors</li> </ul> <p>For applications running on the Bamboo stack, the root cause of these issues is the nature of routing on the Bamboo stack coupled with gradual, horizontal expansion of the routing cluster. On the Cedar stack, the root cause is the fact that Cedar is optimized for concurrent request routing, while some frameworks, like Rails, are not concurrent in their default configurations.</p> <p>We want Heroku to be the best place to build, deploy and scale web and mobile applications. In this case, we’ve fallen short of that promise. We failed to:</p> <ul> <li>Properly document how routing works on the Bamboo stack</li> <li>Understand the service degradation being experienced by our customers and take corrective action</li> <li>Identify and correct confusing metrics reported from the routing layer and displayed by third party tools</li> <li>Clearly communicate the product strategy for our routing service</li> <li>Provide customers with an upgrade path from non-concurrent apps on Bamboo to concurrent Rails apps on Cedar</li> <li>Deliver on the Heroku promise of letting you focus on developing apps while we worry about the infrastructure</li> </ul> <p>We are immediately taking the following actions:</p> <ul> <li>Improving our documentation so that it accurately reflects how our service works across both Bamboo and Cedar stacks</li> <li>Removing incorrect and confusing metrics reported by Heroku or partner services like New Relic</li> <li>Adding metrics that let customers determine queuing impact on application response times</li> <li>Providing additional tools that developers can use to augment our latency and queuing metrics</li> <li>Working to better support concurrent-request Rails apps on Cedar</li> <li>The remainder of this blog post explains the technical details and history of our routing infrastructure, the intent behind the decisions we made along the way, the mistakes we made and what we think is the path forward.</li> </ul> </blockquote>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload