Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Looking at the <a href="https://github.com/richhickey/clojure/blob/59b65669860a1f33825775494809e5d500c19c63/src/clj/clojure/zip.clj#L53" rel="nofollow">source for xml-zip</a>, it doesn't seem like it is <em>entirely</em> lazy:</p> <pre><code>(defn xml-zip "Returns a zipper for xml elements (as from xml/parse), given a root element" {:added "1.0"} [root] (zipper (complement string?) (comp seq :content) (fn [node children] (assoc node :content (and children (apply vector children)))) root)) </code></pre> <p>Note <code>(apply vector children)</code>, which is materializing the <code>children</code> seq to a vector (although it is not materializing the entire descendant tree, so it's still lazy). If you have a very large number of children for a node (e.g., children of <code>&lt;mediawiki&gt;</code>), then even this level of laziness is not enough--<code>:content</code> needs to be a seq too. </p> <p>My knowledge of zippers is extremely limited, so I'm not sure why <code>vector</code> is being used here at all; see if replacing <code>(assoc node :content (and children (apply vector children))))</code> with <code>(assoc node :content children)</code> works, which should keep <code>children</code> as a normal sequence without materializing it.</p> <p>(For that matter, I'm not sure why <code>(apply vector children)</code> instead of <code>(vec children)</code>...)</p> <p><a href="http://clojuredocs.org/clojure_core/clojure.xml/content-handler" rel="nofollow"><code>content-handler</code></a> looks like it is building up all content elements as well in <code>*contents*</code>, so the source of the OOM may be in the content-handler itself.</p> <p>I'm not sure how we can reconcile the zipper interface (tree-like) with the streaming you want. It will work for large xml, but not <em>huge</em> xml.</p> <p>In similar approaches in other languages (e.g. Python's <a href="http://effbot.org/zone/element-iterparse.htm" rel="nofollow">iterparse</a>) a tree is built up iteratively like with zipper. The difference is that the tree will be pruned after successful element processing. </p> <p>For example, in Python with iterparse you would listen for an endElement event on <code>page</code> (i.e. when <code>&lt;/page&gt;</code> occurs in the XML.) At this point you know you have a complete page element which you can process as a tree. After you are finished, you delete the element you just processed and the sibling branches, which controls memory usage.</p> <p>Perhaps you can take this approach here as well. The node provided by the xml zipper is a var to an <code>xml/element</code>. The content handler could return a function that does cleanup on its <code>*current*</code> var when invoked. Then you can call it to prune the tree.</p> <p>Alternatively, you could use SAX "by hand" in clojure for the root element, and create a zipper for each <code>page</code> element as you encounter it.</p>
    singulars
    1. This table or related slice is empty.
    1. POOutOfMemoryError when parsing XML in Clojure with data.zip
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USFrancis Avila
    1. USFrancis Avila
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. CONot sure whether the vector is strictly necessary there, but I don't think it's the cause of the OOM error. The vector is used in the make-node function, which is only called when the zipper is edited in some way. That doesn't appear to be the case here.
      singulars
      1. PO
      1. USAlex
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload