Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>I currently have a project with similar requirements (just more complicated^^).</p> <p>Never spawn a subprocess or thread from your Django view. You have no control of the Django processes and it could be killed, paused etc before the end of the task. It is controlled by the web server (e.g. apache via WSGI).</p> <p>What I would do is an external script, which would run in a separate process. You have two solutions I think :</p> <ul> <li>A process that is always running and crawling the directory where you put your files. It would for example check the directory every ten seconds and process the files</li> <li>Same as above, but run by cron every x seconds. This basically has the same effect</li> <li>Use Celery to create worker processes and add jobs to the queue with your Django application. Then you will need to get the results back by one of the means available with Celery.</li> </ul> <p>Now you probably need to access the information in Django models to email the user in the end. Here you have several solutions :</p> <ul> <li>Import your modules (models etc) from the external script</li> <li>Implement the external script as a custom command (as knutin suggested)</li> <li>Communicate the results to the Django application via a POST request for example. Then you would do the email sending and status changes etc in a normal Django view.</li> </ul> <p>I would go for an external process and import the modules or POST request. This way it is much more flexible. You could for example make use of the multiprocessing module to process several files in the same time (thus using multi-core machines efficiently).</p> <p>A basic workflow would be:</p> <ol> <li>Check the directory for new files</li> <li>For each file (can be parallelized): <ol> <li>Process</li> <li>Send email or notify your Django application </li> </ol></li> <li>Sleep for a while</li> </ol> <p>My project contains really CPU-demanding processing. I currently use an external process that gives processing jobs to a pool of worker processes (that's basically what Celery could do for you) and reports the progress and results back to the Django application via POST requests. It works really well and is relatively scalable, but I will soon change it to use Celery on a cluster.</p>
    singulars
    1. This table or related slice is empty.
    1. PODjango: Should I kick off a separate process?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USMarc Demierre
    1. USMarc Demierre
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThanks for the great feedback. I may need to kick off multiple threads for large files so will look at the multiprocessing module.
      singulars
      1. PO
      1. USFunLovinCoder
    2. COIf the processing is CPU-limited, you have to use processes (e.g. with the processing module) and not threads (threading module). The Python Global Interpreter Lock prevents threads to run truly in parallel, so there is no performance increase (I realized it when doing my project).
      singulars
      1. PO
      1. USMarc Demierre
    3. COAlso, Celery could be the way to go, as it does a lot of the work I described automatically. The only thing is that you have to figure out how to get results as you can't wait for the task to finish (an HTTP callback could do it easily I think).
      singulars
      1. PO
      1. USMarc Demierre
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload