Databases
StackOverflow2013
Postsdbo
POOptimising RSS parsing on App Engine to avoid high CPU warnings
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POOptimising RSS parsing on App Engine to avoid high CPU warnings
    primarykey
    data
    text
    <p>I'm pulling some RSS feeds into a datastore in App Engine to serve up to an iPhone app. I use cron to schedule updating the RSS every x minutes. Each task only parses one RSS feed (which has 15-20 items). I frequently get warnings about high CPU usage in the App Engine dashboard, so I'm looking for ways to optimise my code.</p> <p>Currently, I use minidom (since it's already there on App Engine), but I suspect it's not very efficient!</p> <p>Here's the code: </p> <pre><code> dom = minidom.parseString(urlfetch.fetch(url).content) if dom: items = [] for node in dom.getElementsByTagName('item'): item = RssItem( key_name = self.getText(node.getElementsByTagName('guid')[0].childNodes), title = self.getText(node.getElementsByTagName('title')[0].childNodes), description = self.getText(node.getElementsByTagName('description')[0].childNodes), modified = datetime.now(), link = self.getText(node.getElementsByTagName('link')[0].childNodes), categories = [self.getText(category.childNodes) for category in node.getElementsByTagName('category')] ); items.append(item); db.put(items); def getText(self, nodelist): rc = '' for node in nodelist: if node.nodeType == node.TEXT_NODE: rc = rc + node.data return rc </code></pre> <p>There isn't much going on, but the scripts often take 2-6 seconds CPU time, which seems a bit excessive for looping through 20ish items and reading a few attributes.</p> <p>What can I do to make this faster? Is there anything particularly bad in the above code, or should I change to another way of parsing? Are there are any libraries (that work on App Engine) that would be better, or would I be better parsing the RSS myself?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USDanny Tuppeny
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POOptimising RSS parsing on App Engine to avoid high CPU warnings
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POOptimising RSS parsing on App Engine to avoid high CPU warnings
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POOptimising RSS parsing on App Engine to avoid high CPU warnings
      1. USsystempuntoout
      1. VTFavorite
    1. COHi Danny I updated my comment to include a video by brett slatkin who also had to problem that parsing was to slow and wrote an own lightning fast implementation.
      singulars
      1. POOptimising RSS parsing on App Engine to avoid high CPU warnings
      1. USAlfred
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload