Databases
StackOverflow2013
Postsdbo
POPerformance of mathematical calculations on a text file Java
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPerformance of mathematical calculations on a text file Java
    primarykey
    data
    text
    <p>I'm taking in a text file with around 60,000 lines of point coordinates, (I expect to scale up soon) and performing Mahalanobis distance from each point to every other point, and outputting the result as a text file. That means my results will be nearly 3,600,000,000 lines long. My program creates around 60,000 lines every 1 or two seconds. </p> <p>Am I correct in thinking my code is not able to be multithreaded? Is there a better way to code this algorithm? How do people deal with processes like these?</p> <pre><code>import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.io.IOException; import java.io.PrintWriter; import java.util.ArrayList; import java.util.List; import java.util.Scanner; public class Coord { public int a,b,c,d,e,f; public static void main(String[] args) throws IOException { PrintWriter out = new PrintWriter(new BufferedWriter(new FileWriter("/Users/evanlivingston/2a.txt", true))); Scanner sc = new Scanner(new File("/Users/evanlivingston/1.txt")); List&lt;Coord&gt; coords = new ArrayList&lt;Coord&gt;();{ // for each line in the file while(sc.hasNextLine()) { String[] numstrs = sc.nextLine().split("\\s+"); Coord c = new Coord(); c.a = Integer.parseInt(numstrs[1]); c.b = Integer.parseInt(numstrs[2]); c.c = Integer.parseInt(numstrs[3]); c.d = Integer.parseInt(numstrs[4]); c.e = Integer.parseInt(numstrs[5]); c.f = Integer.parseInt(numstrs[6]); coords.add(c); } // now you have all coords in memory int counter = 0; { for(int i=0; i&lt;coords.size(); i++ ) for( int j=0; j&lt;coords.size(); j++, counter++ ) { Coord c1 = coords.get(i); Coord c2 = coords.get(j); double foo = ((c1.a - c2.a) * (c1.a - c2.a)) *1 ; double goo = ((c1.b - c2.b) * (c1.b - c2.b)) *1 ; double hoo = ((c1.c - c2.c) * (c1.c - c2.c)) *2 ; double joo = ((c1.d - c2.d) * (c1.d - c2.d)) *2 ; double koo = ((c1.e - c2.e) * (c1.e - c2.e)) *4 ; double loo = ((c1.f - c2.f) * (c1.f - c2.f)) *4 ; double zoo = Math.sqrt(foo + goo + hoo + joo + koo + loo); out.println(counter + "; " + i + " " + j + " " + zoo); System.out.println(counter + "; " + i + " " + j + " " + zoo); } out.flush(); out.close(); } } } } </code></pre> <p>My input file looks like</p> <pre><code>0 0 0 0 0 0 0 1 0 0 0 0 0 1 .... 59318 12 2 12 2 12 2 </code></pre> <p>The first number is a place holder. It's a list of all combinations with replacement limited to the amounts you see in the last line.</p> <p>It seems now as though the calculations will take about 16 hours, that still seems too long. Not to mention I estimate the final text output to be around 120 GBs.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USAndrew Thompson
    1. USevanlivingston
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPerformance of mathematical calculations on a text file Java
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPerformance of mathematical calculations on a text file Java
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POPerformance of mathematical calculations on a text file Java
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIt depends on what you want to do. If you could give us more information about your goal we could help you improve your code.
      singulars
      1. POPerformance of mathematical calculations on a text file Java
      1. USdas_weezul
    2. COI bet 2/3 of the time is spent parsing and formatting numbers. If you can arrange it so that you could read binary integers and write binary floats you would a) save time b) probably save space in the output file c) have an output file you culd random access for a specific result.
      singulars
      1. POPerformance of mathematical calculations on a text file Java
      1. USIngo
    3. COInteger.parseInt is quite lame, you could look for a faster implementation.
      singulars
      1. POPerformance of mathematical calculations on a text file Java
      1. USThomas Jungblut
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload