Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Scrapy returns strings in unicode, not ascii. To encode all strings to utf-8, you can write:</p> <pre class="lang-python prettyprint-override"><code>vriskoit['eponimia'] = [s.encode('utf-8') for s in hxs.select('//a[@itemprop="name"]/text()').extract()] </code></pre> <p>But I think that you expect another result. Your code return <strong>one</strong> item with all search results. To return items for each result:</p> <pre class="lang-python prettyprint-override"><code>hxs = HtmlXPathSelector(response) for eponimia, address in zip(hxs.select("//a[@itemprop='name']/text()").extract(), hxs.select("//div[@class='results_address_class']/text()").extract()): vriskoit = VriskoItem() vriskoit['eponimia'] = eponimia.encode('utf-8') vriskoit['address'] = address.encode('utf-8') yield vriskoit </code></pre> <hr> <p><strong>Update</strong></p> <p>JSON exporter writes unicode symbols escaped (e.g. <code>\u03a4</code>) by default, because not all streams can handle unicode. It has option to write them as unicode <code>ensure_ascii=False</code> (see docs for <a href="http://docs.python.org/library/json.html#json.dumps" rel="noreferrer">json.dumps</a>) . But I can't find way to pass this option to standard feed exporter.</p> <p>So if you want exported items to be written in <code>utf-8</code> encoding, e.g. for read them in text editor, you can write custom item pipeline.</p> <p>pipelines.py:</p> <pre class="lang-python prettyprint-override"><code>import json import codecs class JsonWithEncodingPipeline(object): def __init__(self): self.file = codecs.open('scraped_data_utf8.json', 'w', encoding='utf-8') def process_item(self, item, spider): line = json.dumps(dict(item), ensure_ascii=False) + "\n" self.file.write(line) return item def spider_closed(self, spider): self.file.close() </code></pre> <p>Don't forget to add this pipeline to settings.py:</p> <pre class="lang-python prettyprint-override"><code> ITEM_PIPELINES = ['vrisko.pipelines.JsonWithEncodingPipeline'] </code></pre> <p>You can customize pipeline to write data in more human readable format, e.g. you can generate some formated report. <code>JsonWithEncodingPipeline</code> is just basic example.</p>
    singulars
    1. This table or related slice is empty.
    1. POscrapy text encoding
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USreclosedev
    1. USreclosedev
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POscrapy text encoding
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI did what you ve written but i still get the same results: unicode characters. The only way to get utf-8, is to use print vrisko['eponimia'] instead of yield or return.
      singulars
      1. PO
      1. USmindcast
    2. CO@mindcast, Where did you get it? What do you do with items (saving to json feed, csv feed, or maybe custom pipeline)?
      singulars
      1. PO
      1. USreclosedev
    3. COscrapy crawl vrisko -o scraped_data.json -t json or even scrapy crawl vrisko and see the results on my screen. I know i miss something but i can't figure it out. Thank you for your effort.
      singulars
      1. PO
      1. USmindcast
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload