Databases
StackOverflow2013
Postsdbo
POJSON not working in scrapy when calling spider through a python script?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POJSON not working in scrapy when calling spider through a python script?
    primarykey
    data
    text
    <p>When i call my spider through a python script which is as follows:</p> <pre><code>import os os.environ.setdefault('SCRAPY_SETTINGS_MODULE', 'project.settings') from twisted.internet import reactor from scrapy import log, signals from scrapy.crawler import Crawler from scrapy.settings import CrawlerSettings from scrapy.xlib.pydispatch import dispatcher from spiders.image import aqaqspider def stop_reactor(): reactor.stop() dispatcher.connect(stop_reactor, signal=signals.spider_closed) spider = aqaqspider(domain='aqaq.com') crawler = Crawler(CrawlerSettings()) crawler.configure() crawler.crawl(spider) crawler.start() log.start() log.msg('Running reactor...') reactor.run() # the script will block here until the spider is closed log.msg('Reactor stopped.') </code></pre> <p>My Json file is not being created. My pipelines.py is has following code:</p> <pre><code>import json import codecs class JsonWithEncodingPipeline(object): def __init__(self): self.file = codecs.open('scraped_data_utf8.json', 'w', encoding='utf-8') def process_item(self, item, spider): line = json.dumps(dict(item), ensure_ascii=False) + "\n" self.file.write(line) return item def spider_closed(self, spider): self.file.close() </code></pre> <p>When I call my spider with simple command line as scrapy crawl it is working fine i.e JSON file is being created.</p> <p>Please Help me. I am new to scrapy???</p> <p>Thank You all !! I have found the solution....</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USuser2823667
    1. USuser2823667
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POJSON not working in scrapy when calling spider through a python script?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POJSON not working in scrapy when calling spider through a python script?
      1. USRodrigo Andrade
      1. VTFavorite
    1. COI had the same issue didn't get any help http://stackoverflow.com/questions/15483898/calling-scrapy-from-a-python-script-not-creating-json-output-file
      singulars
      1. POJSON not working in scrapy when calling spider through a python script?
      1. USPdksock
    2. CODo you have any idea how to do it ??
      singulars
      1. POJSON not working in scrapy when calling spider through a python script?
      1. USuser2823667
    3. CONo. I guess there is an issue with scrapy itself.
      singulars
      1. POJSON not working in scrapy when calling spider through a python script?
      1. USPdksock
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload