Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>As SteveS said, the <code>csv-to-maps-transformer</code> might try to load the entire file to memory before process it. What you can try to do is split the csv file in smaller parts and send those parts to <code>VM</code> to be processed individually. First, create a component to achieve this first step:</p> <pre><code>public class CSVReader implements Callable{ @Override public Object onCall(MuleEventContext eventContext) throws Exception { InputStream fileStream = (InputStream) eventContext.getMessage().getPayload(); DataInputStream ds = new DataInputStream(fileStream); BufferedReader br = new BufferedReader(new InputStreamReader(ds)); MuleClient muleClient = eventContext.getMuleContext().getClient(); String line; while ((line = br.readLine()) != null) { muleClient.dispatch("vm://in", line, null); } fileStream.close(); return null; } } </code></pre> <p>Then, split your main flow in two</p> <pre><code>&lt;file:connector name="File" workDirectory="yourWorkDirPath" autoDelete="false" streaming="true"/&gt; &lt;flow name="CsvToFile" doc:name="Split and dispatch"&gt; &lt;file:inbound-endpoint path="inboxPath" moveToDirectory="processedPath" pollingFrequency="60000" doc:name="CSV" connector-ref="File"&gt; &lt;file:filename-wildcard-filter pattern="*.csv" caseSensitive="true" /&gt; &lt;/file:inbound-endpoint&gt; &lt;component class="it.aizoon.grpBuyer.AddMessageProperty" doc:name="Add Message Property" /&gt; &lt;component class="com.dgonza.CSVReader" doc:name="Split the file and dispatch every line to VM" /&gt; &lt;/flow&gt; &lt;flow name="storeInDatabase" doc:name="receive lines and store in database"&gt; &lt;vm:inbound-endpoint exchange-pattern="one-way" path="in" doc:name="VM" /&gt; &lt;Choice&gt; . . Your JDBC Stuff . . &lt;Choice /&gt; &lt;/flow&gt; </code></pre> <p>Maintain your current <code>file-connector</code> configuration to enable streaming. With this solution the csv data can be processed without the need to load the entire file to memory first. HTH</p>
    singulars
    1. This table or related slice is empty.
    1. POHow to read huge CSV file in Mule
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USDaniel
    1. USDaniel
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POHow to read huge CSV file in Mule
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COThank you very mutch SteveS and Daniel, I'll try this solution.
      singulars
      1. PO
      1. USParide Letizia
    2. COHi, I tried to use your schema but, ALTHOUGH I'am able to insert some hundreds of rows, at one point I get this message:
      singulars
      1. PO
      1. USParide Letizia
    3. COINFO 2013-05-07 18:23:18,379 [[splitmultithread].FileSplitter.receiver.02] org.mule.transport.file.FileMessageReceiver: Lock obtained on file: C:\workspace_3.4\splitmultithread\src\main\resources\inbox\richiesta.csv ERROR 2013-05-07 18:24:00,144 [[splitmultithread].storeInDatabase.stage1.04] org.mule.processor.AsyncWorkListener: Work caused exception on 'workCompleted'. Work being executed was: org.mule.processor.AsyncInterceptingMessageProcessor$AsyncMessageProcessorWorker@3bc752
      singulars
      1. PO
      1. USParide Letizia
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload