Databases
StackOverflow2013
Postsdbo
POHitting Maximum Recursion Depth Using Pickle / cPickle
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHitting Maximum Recursion Depth Using Pickle / cPickle
    primarykey
    data
    text
    <p>The background: I'm building a trie to represent a dictionary, using a minimal construction algorithm. The input list is 4.3M utf-8 strings, sorted lexicographically. The resulting graph is acyclic and has a maximum depth of 638 nodes. The first line of my script sets the recursion limit to 1100 via <code>sys.setrecursionlimit()</code>.</p> <p>The problem: I'd like to be able to serialize my trie to disk, so I can load it into memory without having to rebuild from scratch (roughly 22 minutes). I have tried both <code>pickle.dump()</code> and <code>cPickle.dump()</code>, with both the text and binary protocols. Each time, I get a stack-trace that looks like the following:</p> <pre class="lang-none prettyprint-override"><code> File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems save(v) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 725, in save_inst save(stuff) File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 648, in save_dict self.memoize(obj) RuntimeError: maximum recursion depth exceeded </code></pre> <p>My data structures are relatively simple: <code>trie</code> contains a reference to a start state, and defines some methods. <code>dfa_state</code> contains a boolean field, a string field, and a dictionary mapping from label to state.</p> <p>I'm not very familiar with the inner workings of <code>pickle</code> - does my max recursion depth need to be greater/equal n times the depth of the trie for some n? Or could this be caused by something else I'm unaware of?</p> <p><strong>Update:</strong> Setting the recursion depth to 3000 didn't help, so this avenue doesn't look promising.</p> <p><strong>Update 2:</strong> You guys were right; I was being short-sighted in assuming that pickle would use a small nesting depth due to default recursion limitations. 10,000 did the trick.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USmartineau
    1. USdanben
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHitting Maximum Recursion Depth Using Pickle / cPickle
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHitting Maximum Recursion Depth Using Pickle / cPickle
      1. USCerin
      1. VTFavorite
    3. VO
      singulars
      1. POHitting Maximum Recursion Depth Using Pickle / cPickle
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload