Databases
StackOverflow2013
Postsdbo
POHow to deserialize 1GB of objects into Python faster than cPickle?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to deserialize 1GB of objects into Python faster than cPickle?
    primarykey
    data
    text
    <p>We've got a Python-based web server that unpickles a number of large data files on startup using <code>cPickle</code>. The data files (pickled using <code>HIGHEST_PROTOCOL</code>) are around 0.4 GB on disk and load into memory as about 1.2 GB of Python objects -- this takes about <strong>20 seconds</strong>. We're using Python 2.6 on 64-bit Windows machines.</p> <p>The bottleneck is certainly not disk (it takes less than 0.5s to actually read that much data), but memory allocation and object creation (there are millions of objects being created). We want to reduce the 20s to decrease startup time.</p> <p>Is there any way to deserialize more than 1GB of objects into Python much faster than <code>cPickle</code> (like 5-10x)? Because the execution time is bound by memory allocation and object creation, I presume using another unpickling technique such as JSON wouldn't help here.</p> <p>I know some interpreted languages have a way to save their entire memory image as a disk file, so they can load it back into memory all in one go, without allocation/creation for each object. Is there a way to do this, or achieve something similar, in Python?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USBen Hoyt
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow to deserialize 1GB of objects into Python faster than cPickle?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow to deserialize 1GB of objects into Python faster than cPickle?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POHow to deserialize 1GB of objects into Python faster than cPickle?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThis might be your chance to get a solid state drive. Is this to speed up dev? To allow you to do quick deployments? Is the lag in reading the data or unpickling it? If you start with an empty instance, what is the start up time?
      singulars
      1. POHow to deserialize 1GB of objects into Python faster than cPickle?
      1. USScott
    2. CONote that I mention in my question the bottleneck is not drive/read speed, but unpickling and object creation speed. It's more for quick deployments -- to allow our server to restart quickly. I'm not quite sure what you mean by "empty instance" here.
      singulars
      1. POHow to deserialize 1GB of objects into Python faster than cPickle?
      1. USBen Hoyt
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload