StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POCalling Scrapy Spider from python script?
text
Body
copied!<p>I have created a spider with name aqaq it is in the file name image.py. the contents of image.py is as follows:</p> <pre><code>from scrapy.spider import BaseSpider from scrapy.selector import HtmlXPathSelector from scrapy.http import Request a=[] from aqaq.items import aqaq import os class aqaqspider(BaseSpider): name = "aqaq" allowed_domains = ["aqaq.com"] start_urls = [ "http://www.aqaq.com/list/female/view-all?limit=all" ] def parse(self, response): hxs = HtmlXPathSelector(response) sites=hxs.select('//ul[@class="list"]/li') for site in sites: name=site.select('a[@class="product-name"]/@href').extract() a.append(name) f=open("url","w+") for i in a: if str(i)=='[]': pass; else: f.write(str(i)[3:-2]+os.linesep) yield Request(str(i)[3:-2].rstrip('\n'),callback=self.parsed) f.close() def parsed(self,response): hxs = HtmlXPathSelector(response) sites=hxs.select('//div[@class="form"]') items=[] for site in sites: item=aqaq() item['title']=site.select('h1/text()').extract() item['cost']=site.select('div[@class="price-container"]/span[@class="regular-price"]/span[@class="price"]/text()').extract() item['desc']=site.select('div[@class="row-block"]/p/text()').extract() item['color']=site.select('div[@id="colours"]/ul/li/a/img/@src').extract() items.append(item) return items </code></pre> <p>i am trying to run this spider with my python script which is as follows:</p> <pre><code>from twisted.internet import reactor from scrapy.crawler import Crawler from scrapy.settings import Settings from scrapy import log, signals from spiders.image import aqaqspider from scrapy.xlib.pydispatch import dispatcher def stop_reactor(): reactor.stop() dispatcher.connect(stop_reactor, signal=signals.spider_closed) spider = aqaqspider(domain='aqaq.com') crawler = Crawler(Settings()) crawler.configure() crawler.crawl(spider) crawler.start()a log.start(loglevel=log.DEBUG) log.msg("------------>Running reactor") result = reactor.run() print result log.msg("------------>Running stoped") </code></pre> <p>while running the above script i am getting the following error:</p> <pre><code>2013-09-27 19:21:06+0530 [aqaq] ERROR: Error downloading <GET http://www.aqaq.com/list/female/view-all?limit=all>: 'Settings' object has no attribute 'overrides' </code></pre> <p>I am a beginner and in need of help ???</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload