Databases
StackOverflow2013
Postsdbo
POhow can I read from a big file and insert it to a database in java in a fast way
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POhow can I read from a big file and insert it to a database in java in a fast way
    primarykey
    data
    text
    <p>I have this text file with this format</p> <pre><code>int | string | string | string | int | string | string | string | int | string | string | string | . . . </code></pre> <p>Size of this file is about 80 MB. I have to read this file and after some evaluation add it to the database. </p> <p>What I do is that I read one line and based on some condition I add them to the database. But this code is taking so long. It's been literally more than a day that I ran this code and no result yet! </p> <p>What can I do to make it faster.</p> <p>I know there should be some way to read the whole file at once.</p> <p>BTW I'm using mysql</p> <p>Help me out guys! </p> <p>Here is my code</p> <pre><code>public void fill_names_db() throws Exception{ MySQLAccess dao = new MySQLAccess(); Scanner stringScanner; BufferedReader in = new BufferedReader(new FileReader("C:\\Users\\havij\\Downloads\\taxdump\\names.dmp")); String tax_id; String name_txt; String unique_name; String name_class; Connection connect=null; connect = dao.newConnection(); while (in.ready()) { String s = in.readLine(); //System.out.println(s); stringScanner = new Scanner(s).useDelimiter("\t|\t"); tax_id = stringScanner.next(); stringScanner.next(); name_txt = stringScanner.next(); stringScanner.next(); unique_name = stringScanner.next(); stringScanner.next(); name_class = stringScanner.next(); if(name_class.equals("scientific name")) dao.insertToDB(connect, "id_to_name", tax_id.toString(), name_txt); if(dao.hasKey(connect,"name_to_id",name_txt)) if (!unique_name.isEmpty()) dao.insertToDB(connect, "name_to_id",unique_name,tax_id.toString(),name_txt,unique_name, name_class ); else if(!name_txt.isEmpty()) dao.insertToDB(connect, "name_to_id",name_txt,tax_id.toString(),name_txt,unique_name, name_class ); } dao.close(connect); in.close(); System.out.println("done"); } </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USAli_IT
    1. USAli_IT
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. CODo you know if you are auto-committing? Batching up calls to the database will help.
      singulars
      1. POhow can I read from a big file and insert it to a database in java in a fast way
      1. USnattyddubbs
    2. CO@nattyddubbs I'm not familiar with auto-committing. I simply just insert a row to a table.
      singulars
      1. POhow can I read from a big file and insert it to a database in java in a fast way
      1. USAli_IT
    3. CO@nattyddubbs it still shouldn't take _that_ long.
      singulars
      1. POhow can I read from a big file and insert it to a database in java in a fast way
      1. USCarsten
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload