StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POimproving performance of writing query results to CSV in java
text
Body
copied!<p>I have the following code that executes a query and writes it directly to a string buffer which then dumps it to a CSV file. I will need to write large amount of records (maximum to a million). This works for a million records it takes about half an hour for a file that is around 200mb! which seems to me like a lot of time, not sure if this is the best. Please recommend me better ways even if it includes using other jars/db connection utils.</p> <pre><code>.... eventNamePrepared = con.prepareStatement(gettingStats + filterOptionsRowNum + filterOptions); ResultSet rs = eventNamePrepared.executeQuery(); int i=0; try{ ...... FileWriter fstream = new FileWriter(realPath + "performanceCollectorDumpAll.csv"); BufferedWriter out = new BufferedWriter(fstream); StringBuffer partialCSV = new StringBuffer(); while (rs.next()) { i++; if (current_appl_id_col_display) partialCSV.append(rs.getString("current_appl_id") + ","); if (event_name_col_display) partialCSV.append(rs.getString("event_name") + ","); if (generic_method_name_col_display) partialCSV.append(rs.getString("generic_method_name") + ","); ..... // 23 more columns to be copied same way to buffer partialCSV.append(" \r\n"); // Writing to file after 10000 records to prevent partialCSV // from going too big and consuming lots of memory if (i % 10000 == 0){ out.append(partialCSV); partialCSV = new StringBuffer(); } } con.close(); out.append(partialCSV); out.close(); </code></pre> <p>Thanks,</p> <p>Tam</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload