Databases
StackOverflow2013
Postsdbo
POMigrate a single threaded app to multi-threaded, parallel execution, monte carlo simulation
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POMigrate a single threaded app to multi-threaded, parallel execution, monte carlo simulation
    text
    copied!<p>I've been tasked with taking an <strong>existing single threaded monte carlo simulation</strong> and <strong>optimising</strong> it. This is a c# console app, no db access it loads data once from a csv file and writes it out at the end, so it's <strong>pretty much just CPU bound</strong>, also only uses about 50mb of memory.</p> <p>I've run it through Jetbrains dotTrace profiler. Of total execution time about 30% is generating uniform random numbers, 24% translating uniform random numbers to normally distributed random numbers.</p> <p>The basic <strong>algorithm is a whole lot of nested for loops</strong>, with random number calls and matrix multiplication at the centre, each iteration returns a double which is added to a results list, this list is periodically sorted and tested for some convergence criteria (at check points every 5% of total iteration count) if acceptable the program breaks out of the loops and writes the results, else it proceeds to the end.</p> <p>I'd like developers to weigh in on:</p> <ul> <li>should I use <strong>new Thread v ThreadPool</strong></li> <li>should I look at the <strong>Microsoft Parallels Extension library</strong></li> <li>should I look at <strong>AForge.Net Parallel.For</strong>, <a href="http://code.google.com/p/aforge/" rel="nofollow noreferrer">http://code.google.com/p/aforge/</a> any other libraries?</li> </ul> <p>Some <strong>links to tutorials</strong> on the above would be most welcome as <strong>I've never written any parallel or multi-threaded code</strong>.</p> <ul> <li>best strategies for generating en-mass normally distributed random numbers, and then consuming these. Uniform random numbers are never used in this state by the app, they are always translated to <strong>normally distributed</strong> and then consumed.</li> <li>good fast libraries (parallel?) for random number generation</li> <li><strong>memory considerations as I take this parallel</strong>, how much extra will I require.</li> </ul> <p>Current app takes 2 hours for 500,000 iterations, business needs this to scale to 3,000,000 iterations and be called mulitple times a day so need some heavy optimisation.</p> <p><strong>Particulary would like to hear from people</strong> who have used <strong>Microsoft Parallels Extension</strong> or <strong>AForge.Net Parallel</strong></p> <p>This needs to be productionised fairly quickly so <strong>.net 4 beta is out</strong> even though I know it has concurrency libraries baked in, we can look at migrating to .net 4 later down the track once it's released. For the moment the server has .Net 2, I've submitted for review an upgrade to .net 3.5 SP1 which my dev box has.</p> <p>Thanks</p> <p><strong>Update</strong></p> <p>I've just tried the Parallel.For implementation but it comes up with some weird results. Single threaded:</p> <pre><code>IRandomGenerator rnd = new MersenneTwister(); IDistribution dist = new DiscreteNormalDistribution(discreteNormalDistributionSize); List&lt;double&gt; results = new List&lt;double&gt;(); for (int i = 0; i &lt; CHECKPOINTS; i++) { results.AddRange(Oblist.Simulate(rnd, dist, n)); } </code></pre> <p>To:</p> <pre><code>Parallel.For(0, CHECKPOINTS, i =&gt; { results.AddRange(Oblist.Simulate(rnd, dist, n)); }); </code></pre> <p>Inside simulate there are many calls to rnd.nextUniform(), <strong>I think I am getting many values that are the same</strong>, is this likely to happen because this is now parallel?</p> <p>Also maybe issues with the List AddRange call not being thread safe? I see this </p> <p>System.Threading.Collections.BlockingCollection might be worth using, but it only has an Add method no AddRange so I'd have to look over there results and add in a thread safe manner. Any insight from someone who has used Parallel.For much appreciated. I switched to the <strong>System.Random</strong> for my calls temporarily as I was getting an exception when calling nextUniform with my Mersenne Twister implementation, <strong>perhaps it wasn't thread safe</strong> a certain array was getting an <strong>index out of bounds</strong>....</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload