StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POIs there a way to circumvent Python list.append() becoming progressively slower in a loop as the list grows?
text
Body
copied!<p>I have a big file I'm reading from, and convert every few lines to an instance of an Object.</p> <p>Since I'm looping through the file, I stash the instance to a list using list.append(instance), and then continue looping.</p> <p>This is a file that's around ~100MB so it isn't too large, but as the list grows larger, the looping slows down progressively. (I print the time for each lap in the loop).</p> <p>This is not intrinsic to the loop ~ when I print every new instance as I loop through the file, the program progresses at constant speed ~ it is only when I append them to a list it gets slow.</p> <p>My friend suggested disabling garbage collection before the while loop and enabling it afterward & making a garbage collection call.</p> <p>Did anyone else observe a similar problem with list.append getting slower? Is there any other way to circumvent this?</p> <hr> <p>I'll try the following two things suggested below.</p> <p>(1) "pre-allocating" the memory ~ what's the best way to do this? (2) Try using deque</p> <p>Multiple posts (see comment by Alex Martelli) suggested memory fragmentation (he has a large amount of available memory like I do) ~ but no obvious fixes to performance for this.</p> <p>To replicate the phenomenon, please run the test code provided below in the answers and assume that the lists have useful data.</p> <hr> <p>gc.disable() and gc.enable() helps with the timing. I'll also do a careful analysis of where all the time is spent.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload