Databases
StackOverflow2013
Postsdbo
POJava Desktop Application to Upload CSV Files to Database
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POJava Desktop Application to Upload CSV Files to Database
    primarykey
    data
    text
    <p>I'm working on desktop application (Extract Transform Load type of app) that requires uploading of large CSV files (about 6 GB) to a database using Java as front end and Oracle as a backend but I'm open to other databases and tools as well (i.e. SQLite, H2, HSQLDB, MySQL).</p> <p>I tried using OpenSource Java libraries for parsing and uploading CSV files such as:</p> <ol> <li><a href="http://opencsv.sourceforge.net/" rel="nofollow">OpenCSV</a> </li> <li><a href="http://supercsv.sourceforge.net/" rel="nofollow">Super CSV</a></li> </ol> <p>The problem with these libraries is efficiency. They require too much time for uploading. For example, It'll take 6 hours to upload 4 Gb of CSV file.</p> <p>Having this experience, I decided to try database utilities for uploading CSV files. One of them is Oracle SQL*Loader. It's faster and offers desirable result. It can remove whitespaces, load CSV files to various tables, use oracle functions suchs as decode, etc. </p> <p>However, I have the following problems:</p> <ol> <li><p>I couldn't find a way to get SQL*Loader's errors/logs in Java and display them to the user.</p></li> <li><p>Since, SQL*Loader is proprietary, there's no way to embed it to my application installation. For example, the user needs to download and install Oracle Client separately before installing my application. Another thing is I need to check if Oracle Client is installed before installation process begins.</p></li> </ol> <p>I looked for other probable solutions and found headless databases like SQLite and H2.</p> <p>On SQLite, I use the .import command to upload CSV files. But there's no way I could call it in Java.</p> <p>I turn to H2 database but I have hard time uploading since it does not support escaping the header row.</p> <p>Please let me know your suggestions.</p> <p>Thanks you.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USepsac
    1. USepsac
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POJava Desktop Application to Upload CSV Files to Database
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POJava Desktop Application to Upload CSV Files to Database
      1. USAdam Smith
      1. VTFavorite
    1. CODo you have to use SQLite? Can you not use external tables to load the data? Excluding headers and footer is easy using external tables.
      singulars
      1. POJava Desktop Application to Upload CSV Files to Database
      1. USOllie
    2. CONot at all. I'm creating a *java* desktop application that could load a *large .csv* file to my local database as fast as possible regardless of the database.
      singulars
      1. POJava Desktop Application to Upload CSV Files to Database
      1. USepsac
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload