Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Just to be clear, <code>(:content (data.xml/parse rdr :coalescing false))</code> IS lazy. Check its class or pull the first item (it will return instantly) if you're not convinced.</p> <p>That said, a couple things to watch out for when processing large sequences: holding onto the head, and unrealized/nested laziness. I think your code suffers from the latter.</p> <p>Here's what I recommend:</p> <p>1) Add <code>(dorun)</code> to the end of the <code>-&gt;&gt;</code> chain of calls. This will force the sequence to be fully realized without holding onto the head.</p> <p>2) Change <code>for</code> in <code>process-page</code> to <code>doseq</code>. You're spitting to a file, which is a side effect, and you don't want to do that lazily here.</p> <p>As Arthur recommends, you may want to open an output file once and keep writing to it, rather than opening &amp; writing (spit) for every Wikipedia entry.</p> <p><strong>UPDATE</strong>:</p> <p>Here's a rewrite which attempts to separate concerns more clearly:</p> <pre><code>(defn filter-tag [tag xml] (filter #(= tag (:tag %)) xml)) ;; lazy (defn revision-seq [xml] (for [page (filter-tag :page (:content xml)) :let [title (article-title page)] revision (filter-tag :revision (:content page)) :let [user (revision-user revision) time (revision-timestamp revision)]] [time user title])) ;; eager (defn transform [in out] (with-open [r (io/input-stream in) w (io/writer out)] (binding [*out* out] (let [xml (data.xml/parse r :coalescing false)] (doseq [[time user title] (revision-seq xml)] (println (str "\"" time "\";\"" user "\";\"" title "\"\n"))))))) (transform "dump.xml" "data.csv") </code></pre> <p>I don't see anything here that would cause excessive memory use.</p>
    singulars
    1. This table or related slice is empty.
    1. POHuge file in Clojure and Java heap space error
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USJustin Kramer
    1. USJustin Kramer
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POHuge file in Clojure and Java heap space error
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COThe point about dorun could be made a little clearer for someone new to Clojure: the open-file function as shown in the question returns the sequence of results of calls to process-pages, and when the function is called from the repl, printing the sequence causes all the results to be held in memory at the same time. Calling dorun on the result causes the elements of the sequence to be evaluated and nil to be returned, so that there is never a need to have all the results in memory at the same time.
      singulars
      1. PO
      1. USJouni K. Seppänen
    2. COThanx for the explanation! I do understand (hopefully) now how the laziness works in this code snippet and changed what you proposed, but still `OutOfMemoryError: Java heap space`. I'm working on a 1GB sample of the final file, but it still kicks the memory error. Would be really thankful for any help.
      singulars
      1. PO
      1. UStrzewiczek
    3. COSee my latest update. If you still get OutOfMemory error, I'm not sure why. I used code very similar to this with no memory problems.
      singulars
      1. PO
      1. USJustin Kramer
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload