Databases
StackOverflow2013
Postsdbo
POAccess Django models with scrapy: defining path to Django project
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POAccess Django models with scrapy: defining path to Django project
    text
    copied!<p>I'm very new to Python and Django. I'm currently exploring using Scrapy to scrape sites and save data to the Django database. My goal is to run a spider based on domain given by a user. </p> <p>I've written a spider that extracts the data I need and store it correctly in a json file when calling </p> <pre><code>scrapy crawl spider -o items.json -t json </code></pre> <p>As described in the <a href="http://doc.scrapy.org/en/latest/intro/tutorial.html" rel="nofollow noreferrer">scrapy tutorial</a>.</p> <p>My goal is now to get the spider to succesfully to save data to the Django database, and then work on getting the spider to run based on user input. </p> <p>I'm aware that various posts exists on this subject, such as these: <a href="https://stackoverflow.com/questions/4271975/access-django-models-inside-of-scrapy">link 1</a> <a href="https://stackoverflow.com/questions/7883196/saving-django-model-from-scrapy-project">link 2</a> <a href="https://stackoverflow.com/questions/4271975/access-django-models-inside-of-scrapy">link 3</a></p> <p>But having spend more than 8 hours on trying to get this to work, I'm assuming i'm not the only one still facing issues with this. I'll therefor try and gather all the knowledge i've gotten so far in this post, as well a hopefully posting a working solution at a later point. Because of this, this post is rather long. </p> <p>It appears to me that there is two different solutions to saving data to the Django database from Scrapy. One is to use <a href="https://scrapy.readthedocs.org/en/latest/topics/djangoitem.html" rel="nofollow noreferrer">DjangoItem</a>, another is to to import the models directly(as done <a href="https://stackoverflow.com/questions/7883196/saving-django-model-from-scrapy-project">here</a>).</p> <p>I'm not completely aware of the advantages and disadvantages of these two, but it seems like the difference is simply the using DjangoItem is just more convenient and shorter.</p> <p><strong>What i've done:</strong> </p> <p>I've added:</p> <pre><code>def setup_django_env(path): import imp, os from django.core.management import setup_environ f, filename, desc = imp.find_module('settings', [path]) project = imp.load_module('settings', f, filename, desc) setup_environ(project) setup_django_env('/Users/Anders/DjangoTraining/wsgi/') </code></pre> <p><strong>Error i'm getting is:</strong></p> <pre><code>ImportError: No module named settings </code></pre> <p>I'm thinking i'm defining the path to my Django project in a wrong way?</p> <p>I've also tried the following:</p> <pre><code>setup_django_env('../../') </code></pre> <p>How do I define the path to my Django project correctly? (if that is the issue)</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload