Databases
StackOverflow2013
Postsdbo
POAppengine app is 404ing on /_ah/warmup out of nowhere; spinning up useless instances and causing a lot of cost
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POAppengine app is 404ing on /_ah/warmup out of nowhere; spinning up useless instances and causing a lot of cost
    primarykey
    data
    text
    <p>I noticed that the App Engine mailing list suggested I direct my problems to Stack Overflow, so I am filing my bug report here.</p> <p>Starting at 2013-06-25 23:05:40.756 US/Pacific my app behavior significantly changed without any new deployments or other state modification. A request from an iPhone client to <code>/</code> caused an <code>/_ah/warmup</code> request that <code>404</code>'d and began an unrelenting set of cascading failures which are still occurring when I enable the app. Each new <code>404</code> is causing my app to spin up new instances which in turn <code>404</code> on <code>/_ah/warmup</code>, causing a cascade of unneeded instances <strong>which have sucked up my entire daily budget and caused downtime on my site for over 16 hours!</strong></p> <p>The application is currently disabled until I can resolve the issue, as it is just sucking up cost for no reason and spinning up dozens of unneeded instances in a matter of minutes.</p> <p>In the last 24 hours I can see ~4 requests from non <code>/_ah/warmup/</code> endpoints, and <strong>roughly 3k failing requests from <code>/_ah/warmup</code>!</strong></p> <p>Again, to reiterate, my co-founder and I have made 0 code modifications to the site. I don't even believe we've done any new datastore writes in the last 24 hours. This sudden <code>/_ah/warmup</code> failure has come seemingly out of nowhere.</p> <p>I did have <code>warmup</code> enabled by adding it to <code>inbound_services</code> in <code>app.yaml</code>. I tried redeploying without any mention of <code>inbound_services</code> but that did not fix things.</p> <p>I've also tried adding an <code>/_ah/warmup</code> handler directly into my main bottle app, but this did not get picked up it seems and the instances still failover. This was true regardless of whether or not <code>inbound_services</code> was enabled with <code>warmup</code> in my <code>app.yaml</code>.</p> <p>I cannot reproduce any of this locally and it has come seemingly out of nowhere. It feels as though a recent appengine runtime update clobbered something I'm not aware of. Any help would be much appreciated; this seems to be the only place I can file issues such as this for the entirety of the App Engine platform.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USmvanveen
    1. USmvanveen
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POAppengine app is 404ing on /_ah/warmup out of nowhere; spinning up useless instances and causing a lot of cost
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POAppengine app is 404ing on /_ah/warmup out of nowhere; spinning up useless instances and causing a lot of cost
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThere were some anomalies yesterday.. the issue is still happening ? https://code.google.com/status/appengine
      singulars
      1. POAppengine app is 404ing on /_ah/warmup out of nowhere; spinning up useless instances and causing a lot of cost
      1. USNix
    2. COYes I have disabled the app until I can stop the cascade of failing instances that are sucking up cost for no reason, and my site is currently down!
      singulars
      1. POAppengine app is 404ing on /_ah/warmup out of nowhere; spinning up useless instances and causing a lot of cost
      1. USmvanveen
    3. COstill happening ? Have you tried killin the instances?
      singulars
      1. POAppengine app is 404ing on /_ah/warmup out of nowhere; spinning up useless instances and causing a lot of cost
      1. USNix
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload