Databases
StackOverflow2013
Postsdbo
POmarshal dumps faster, cPickle loads faster
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POmarshal dumps faster, cPickle loads faster
    primarykey
    data
    text
    <p>I'm implementing a program that needs to serialize and deserialize large objects, so I was making some tests with <code>pickle</code>, <code>cPickle</code> and <code>marshal</code> modules to choose the best module. Along the way I found something very interesting:</p> <p>I'm using <code>dumps</code> and then <code>loads</code> (for each module) on a list of dicts, tuples, ints, float and strings.</p> <p>This is the output of my benchmark:</p> <pre><code>DUMPING a list of length 7340032 ---------------------------------------------------------------------- pickle =&gt; 14.675 seconds length of pickle serialized string: 31457430 cPickle =&gt; 2.619 seconds length of cPickle serialized string: 31457457 marshal =&gt; 0.991 seconds length of marshal serialized string: 117440540 LOADING a list of length: 7340032 ---------------------------------------------------------------------- pickle =&gt; 13.768 seconds (same length?) 7340032 == 7340032 cPickle =&gt; 2.038 seconds (same length?) 7340032 == 7340032 marshal =&gt; 6.378 seconds (same length?) 7340032 == 7340032 </code></pre> <p>So, from these results we can see that <code>marshal</code> was extremely fast in the <strong>dumping</strong> part of the benchmark:</p> <blockquote> <p>14.8x times faster than <code>pickle</code> and 2.6x times faster than <code>cPickle</code>.</p> </blockquote> <p>But, for my big surprise, <code>marshal</code> was by far slower than <code>cPickle</code> in the <strong>loading</strong> part:</p> <blockquote> <p>2.2x times faster than <code>pickle</code>, but 3.1x times slower than <code>cPickle</code>.</p> </blockquote> <p>And as for RAM, <code>marshal</code> performance while <strong>loading</strong> was also very inefficient:</p> <p><img src="https://i.stack.imgur.com/ZAFoV.png" alt="Ubuntu System Monitor"></p> <p>I'm guessing the reason why loading with <code>marshal</code> is so slow is somehow related with the length of the its serialized string (much longer than <code>pickle</code> and <code>cPickle</code>).</p> <ul> <li>Why <code>marshal</code> dumps faster and loads slower?</li> <li>Why <code>marshal</code> serialized string is so long?</li> <li>Why <code>marshal</code>'s loading is so inefficient in RAM?</li> <li>Is there a way to improve <code>marshal</code>'s loading performance?</li> <li>Is there a way to merge <code>marshal</code> fast dumping with <code>cPickle</code> fast loading?</li> </ul>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USjuliomalegria
    1. USjuliomalegria
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POmarshal dumps faster, cPickle loads faster
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POmarshal dumps faster, cPickle loads faster
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POmarshal dumps faster, cPickle loads faster
      1. This table or related slice is empty.
      1. VTDownMod
    1. COdownvoter, care to share?
      singulars
      1. POmarshal dumps faster, cPickle loads faster
      1. USjuliomalegria
    2. COYour question is a dead-end. The `marshal` module is not meant to be used as an alternative to `pickle`. There is no official documentation for the marshal file format and it might change from version to version, so your benchmark results might be false in the future.
      singulars
      1. POmarshal dumps faster, cPickle loads faster
      1. USFerdinand Beyer
    3. COConcerning the speed differences: I suspect it's all about file IO: The file produced by marshal is nearly four times as large (112MB vs 30MB).
      singulars
      1. POmarshal dumps faster, cPickle loads faster
      1. USFerdinand Beyer
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload