Databases
StackOverflow2013
Postsdbo
POProblems during counting strings in the txt file
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POProblems during counting strings in the txt file
    primarykey
    data
    text
    <p>I am developing a progam which reads a text file and creates a report. The content of the report is the following: the number of every string in file, its "status", and some symbols of every string beginning. It works well with file up to 100 Mb. </p> <p>But when I run the program with input files which are bigger than 1,5Gb in size and contain more than 100000 lines, I get the following error:</p> <pre><code>&gt; Exception in thread "main" java.lang.OutOfMemoryError: Java heap space &gt; at java.util.Arrays.copyOfRange(Unknown Source) at &gt; java.lang.String.&lt;init&gt;(Unknown Source) at &gt; java.lang.StringBuffer.toString(Unknown Source) at &gt; java.io.BufferedReader.readLine(Unknown Source) at &gt; java.io.BufferedReader.readLine(Unknown Source) at &gt; org.apache.commons.io.IOUtils.readLines(IOUtils.java:771) at &gt; org.apache.commons.io.IOUtils.readLines(IOUtils.java:723) at &gt; org.apache.commons.io.IOUtils.readLines(IOUtils.java:745) at &gt; org.apache.commons.io.FileUtils.readLines(FileUtils.java:1512) at &gt; org.apache.commons.io.FileUtils.readLines(FileUtils.java:1528) at &gt; org.apache.commons.io.ReadFileToListSample.main(ReadFileToListSample.java:43) </code></pre> <p>I increased VM arguments up to -Xms128m -Xmx1600m (in eclipse run configuration) but this did not help. Specialists from OTN forum advised me to read some books and improve my program's performance. Could anybody help me to improve it? Thank you.</p> <p>code:</p> <pre><code>import org.apache.commons.io.FileUtils; import java.io.File; import java.io.FileNotFoundException; import java.io.FileOutputStream; import java.io.FileReader; import java.io.IOException; import java.io.LineNumberReader; import java.io.PrintStream; import java.util.List; public class ReadFileToList { public static void main(String[] args) throws FileNotFoundException { File file_out = new File ("D:\\Docs\\test_out.txt"); FileOutputStream fos = new FileOutputStream(file_out); PrintStream ps = new PrintStream (fos); System.setOut (ps); // Create a file object File file = new File("D:\\Docs\\test_in.txt"); FileReader fr = null; LineNumberReader lnr = null; try { // Here we read a file, sample.txt, using FileUtils // class of commons-io. Using FileUtils.readLines() // we can read file content line by line and return // the result as a List of string. List&lt;String&gt; contents = FileUtils.readLines(file); // // Iterate the result to print each line of the file. fr = new FileReader(file); lnr = new LineNumberReader(fr); for (String line : contents) { String begin_line = line.substring(0, 38); // return 38 chars from the string String begin_line_without_null = begin_line.replace("\u0000", " "); String begin_line_without_null_spaces = begin_line_without_null.replaceAll(" +", " "); int stringlenght = line.length(); line = lnr.readLine(); int line_num = lnr.getLineNumber(); String status; // some correct length for if int c_u_length_f = 12; int c_ea_length_f = 13; int c_a_length_f = 2130; int c_u_length_e = 3430; int c_ea_length_e = 1331; int c_a_length_e = 442; int h_ext = 6; int t_ext = 6; if ( stringlenght == c_u_length_f || stringlenght == c_ea_length_f || stringlenght == c_a_length_f || stringlenght == c_u_length_e || stringlenght == c_ea_length_e || stringlenght == c_a_length_e || stringlenght == h_ext || stringlenght == t_ext) status = "ok"; else status = "fail"; System.out.println(+ line_num + stringlenght + status + begin_line_without_null_spaces); } } catch (IOException e) { e.printStackTrace(); } } } </code></pre> <p>Also specialists from OTN said that this programm opens the input and reading it twice. May be some mistakes in "for statement"? But I can't find it. Thank you.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USanimuson
    1. USMay12
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POProblems during counting strings in the txt file
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    1. COHow about not reading in the entire file at once?
      singulars
      1. POProblems during counting strings in the txt file
      1. USDave Newton
    2. COthe code 'line = lnr.readLine;' is where you are reading the line a second time. You're already in the foreach loop with lines read from 'content'.
      singulars
      1. POProblems during counting strings in the txt file
      1. USJustin
    3. COI would break the file into sections if it is a large file, if you're getting an OutOfMemoryError then obviously the file you are loading is too large.
      singulars
      1. POProblems during counting strings in the txt file
      1. USJohn
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload