Databases
StackOverflow2013
Postsdbo
POSpring Integration 1.0 RC2: Streaming file content?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSpring Integration 1.0 RC2: Streaming file content?
    primarykey
    data
    text
    <p>I've been trying to find information on this, but due to the immaturity of the Spring Integration framework I haven't had much luck.</p> <p>Here is my desired work flow:</p> <ol> <li><p>New files are placed in an 'Incoming' directory</p></li> <li><p>Files are picked up using a file:inbound-channel-adapter</p></li> <li><p>The file content is streamed, N lines at a time, to a 'Stage 1' channel, which parses the line into an intermediary (shared) representation. </p></li> <li><p>This parsed line is routed to multiple 'Stage 2' channels.</p></li> <li><p>Each 'Stage 2' channel does its own processing on the N available lines to convert them to a final representation. This channel must have a queue which ensures no Stage 2 channel is overwhelmed in the event that one channel processes significantly slower than the others.</p></li> <li><p>The final representation of the N lines is written to a file. There will be as many output files as there were routing destinations in step 4.</p></li> </ol> <p>**'N' above stands for any reasonable number of lines to read at a time, from [1, whatever I can fit into memory reasonably], but is guaranteed to always be less than the number of lines in the full file.*</p> <p>How can I accomplish <strong>streaming (steps 3, 4, 5)</strong> in Spring Integration? It's fairly easy to do without streaming the files, but my files are large enough that I cannot read the entire file into memory.</p> <p>As a side note, I have a working implementation of this work flow without Spring Integration, but since we're using Spring Integration in other places in our project, I'd like to try it here to see how it performs and how the resulting code compares for length and clarity.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USgdm
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSpring Integration 1.0 RC2: Streaming file content?
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload