Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>First of all, do you have any possibility to use Java 7? There you have a <code>FileVisitor</code> and the <code>Files.walkFileTree</code>, which should probably work within your memory constraints.</p> <p><strike>Otherwise, the only way I can think of is to use <code>File.listFiles(FileFilter filter)</code> with a filter that always returns <code>false</code> (ensuring that the full array of files is never kept in memory), but that catches the files to be processed along the way, and perhaps puts them in a producer/consumer queue or writes the file-names to disk for later traversal.</p> <p>Alternatively, if you control the names of the files, or if they are named in some nice way, you could process the files in chunks using a filter that accepts filenames on the form <code>file0000000</code>-<code>filefile0001000</code> then <code>file0001000</code>-<code>filefile0002000</code> and so on.</p> <p>If the names are <em>not</em> named in a nice way like this, you could try filtering them based on the hash-code of the file-name, which is supposed to be fairly evenly distributed over the set of integers.</strike></p> <hr> <p><strong>Update:</strong> Sigh. Probably won't work. Just had a look at the listFiles implementation:</p> <pre><code>public File[] listFiles(FilenameFilter filter) { String ss[] = list(); if (ss == null) return null; ArrayList v = new ArrayList(); for (int i = 0 ; i &lt; ss.length ; i++) { if ((filter == null) || filter.accept(this, ss[i])) { v.add(new File(ss[i], this)); } } return (File[])(v.toArray(new File[v.size()])); } </code></pre> <p>so it will probably fail at the first line anyway... Sort of disappointing. I believe your best option is to put the files in different directories.</p> <p>Btw, could you give an example of a file name? Are they "guessable"? Like</p> <pre><code>for (int i = 0; i &lt; 100000; i++) tryToOpen(String.format("file%05d", i)) </code></pre>
    singulars
    1. This table or related slice is empty.
    1. POHow to list a 2 million files directory in java without having an "out of memory" exception
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USaioobe
    1. USaioobe
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POHow to list a 2 million files directory in java without having an "out of memory" exception
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COJava 7 is not an option right now. Currently i'm trying the filter option. Thankfully the files have a hierarchy written in the filename. So this option could work.
      singulars
      1. PO
      1. USFgblanch
    2. COaioobe effectively it didn't work. I've found the filenames are "guessables" :) so i'll do it the other way around: Generate the filenames and then go to the folder and try to reach them. Thanks a lot for your help
      singulars
      1. PO
      1. USFgblanch
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload