StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POSpring Batch Multi Threading - How to make each thread read unique records?
text
Body
copied!<p>This question has been asked a lot of times in many forums. But I don't see an answer that fits me. I am trying to implement Multi-threaded Step in my spring batch implementation. </p> <ol> <li><p>Have a staging table with 100k records </p></li> <li><p>Want to process it in 10 threads of commit interval 300 per thread- so 3000 records at any point of time. </p></li> <li><p>I defined a task executor and referred it inside the step i wanted to multi thread </p></li> <li><p>My idea is that first I would get the thread pool size (10) and update the thread_id column with a velue(can be 1-10) to each of the 100k records. In this case of 10 threads and 100k records so 10k records will be assigned one id - I am trying to implement a stagingsteplistener to do this. </p></li> <li><p>wrote a reader for this staging table. task executor will create 10 readers and each reader must read 300 different records and process them - Now how do I pass a common id between the step listener and reader so that each thread will have its own set of records to process.</p></li> </ol> <p>As of now I have only one JVM. So I am thinking of doing this in Multi Threaded step itself rather than thinking about partition based approach.</p> <p>Please help......</p> <p>I referred pro spring batch book and created a staging step listener which is accepting a run id from the job configuration xml using job parameters as below</p> <pre><code><beans:bean id="stagingStepListener" class="com.apress.springbatch.statement.listener.StagingStepListener" scope="step"> <beans:property name="dataSource" ref="dataSource"/> <beans:property name="tableName" value="transaction"/> <beans:property name="whereClause" value="where jobId is null and processed is null"/> <beans:property name="jobId" value="#{jobParameters[run.id]}"/> </beans:bean> </code></pre> <p>What i dont find is this? From where is this "run.id" coming from. I dont see that in any place in the book. I copied the same implementation in my spring batch and when i run it I see exception saying that run.id is not identifiable. Please help me about how to do this?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload