Databases
StackOverflow2013
Postsdbo
POR code slowing with increased iterations
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POR code slowing with increased iterations
    primarykey
    data
    text
    <p>I've been trying to increase the speed of some code. I've removed all loops, am using vectors and have streamed lined just about everything. I've timed each iteration of my code and it appears to be slowing as iterations increase. </p> <pre><code>### The beginning iterations user system elapsed 0.03 0.00 0.03 user system elapsed 0.03 0.00 0.04 user system elapsed 0.03 0.00 0.03 user system elapsed 0.04 0.00 0.05 ### The ending iterations user system elapsed 3.06 0.08 3.14 user system elapsed 3.10 0.05 3.15 user system elapsed 3.08 0.06 3.15 user system elapsed 3.30 0.06 3.37 </code></pre> <p>I have 598 iterations and right now it takes about 10 minutes. I'd like to speed things up. Here's how my code looks. You'll need the <code>RColorBrewer</code> and <code>fields</code> packages. Here's my <a href="https://drive.google.com/file/d/0B__ibLQJIWjbbHlITkFqbEJsd0E/edit?usp=sharing" rel="nofollow noreferrer">data</a>. Yes I know its big, make sure you download the zip file. </p> <pre><code> StreamFlux &lt;- function(data,NoR,NTS){ ###Read in data to display points### WLX = c(8,19,29,20,13,20,21) WLY = c(25,28,25,21,17,14,12) WLY = 34 - WLY WLX = WLX / 44 WLY = WLY / 33 timedata = NULL mf &lt;- function(i){ b = (NoR+8) * (i-1) + 8 ###I read in data one section at a time to avoid headers mydata = read.table(data,skip=b,nrows=NoR, header=FALSE) rows = 34-mydata[,2] cols = 45-mydata[,3] flows = mydata[,7] rows = as.numeric(rows) cols = as.numeric(cols) rm(mydata) ###Create Flux matrix flow_mat &lt;- matrix(0,44,33) ###Populate matrix### flow_mat[(rows - 1) * 44 + (45-cols)] &lt;- flows+flow_mat[(rows - 1) * 44 + (45-cols)] flow_mat[flow_mat == 0] &lt;- NA rm(flows) rm(rows) rm(cols) timestep = i ###Specifying jpeg info### jpeg(paste("Steamflow", timestep, ".jpg",sep = ''), width = 640, height=441,quality=75,bg="grey") image.plot(flow_mat, zlim=c(-1,1), col=brewer.pal(11, "RdBu"),yaxt="n", xaxt="n", main=paste("Stress Period ", timestep, sep = "")) points(WLX,WLY) dev.off() rm(flow_mat) } ST&lt;- function(x){functiontime=system.time(mf(x)) print(functiontime)} lapply(1:NTS, ST) } </code></pre> <h3>This is how to run the function</h3> <pre><code>###To run all timesteps### StreamFlux("stream_out.txt",687,598) ###To run the first 100 timesteps### StreamFlux("stream_out.txt",687,100) ###The first 200 timesteps### StreamFlux("stream_out.txt",687,200) </code></pre> <h3>To test remove <code>print(functiontime)</code> to stop it printing at every timestep then</h3> <pre><code>&gt; system.time(StreamFlux("stream_out.txt",687,100)) user system elapsed 28.22 1.06 32.67 &gt; system.time(StreamFlux("stream_out.txt",687,200)) user system elapsed 102.61 2.98 106.20 </code></pre> <p>What I'm looking for is anyway to speed up running this code and possibly an explanation of why it is slowing down? Should I just run it in parts, seems a stupid solution. I've read about <code>dlply</code> from the <code>plyr</code>. It seems to have worked <a href="https://stackoverflow.com/questions/15052578/lapply-slows-with-increasing-loopspackage">here</a> but would that help in my case? How about parallel processing, I think I can figure that out but is it worth the trouble in this case?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USCCurtis
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POR code slowing with increased iterations
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POR code slowing with increased iterations
      1. This table or related slice is empty.
      1. VTDownMod
    3. VO
      singulars
      1. POR code slowing with increased iterations
      1. This table or related slice is empty.
      1. VTDownMod
    1. COdo some work yourself first and distill your problem to a **minimal** example where you're having problems
      singulars
      1. POR code slowing with increased iterations
      1. USeddi
    2. COI'm not going to spend any time on an example this large and complicated. But give you two pieces of advice: (1) Use `RProf` to determine _exactly_ which pieces are the bottleneck, (2) I'll bet that your strategy of read successive portions from disk is very sub-optimal. Touching the (presumably spinning) disk is slow. Read all the data in at once. Even then, you're trying to write several hundred plots to disk. That also won't be screaming fast, depending on your hard drive.
      singulars
      1. POR code slowing with increased iterations
      1. USjoran
    3. CO@eddi I've been working to improve this code for some time now. If I knew what the source of my problems was I would have posted a minimal example as I have done for other sections of my code when I was in the process of streamlining things. I'm not sure where the problem is now so i wanted to give everyone a look at the full code. What is so complicated about what I posted? All I'm doing is formatting data, plotting it and repeating with `lapply`. The function ST isn't even needed. Its just there to print run times.
      singulars
      1. POR code slowing with increased iterations
      1. USCCurtis
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload