Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>A lot of these answers are just wild guesses. I like Mike Graham's the best because he's right about how lists are implemented. But I've written some code to reproduce your claim and look into it further. Here are some findings.</p> <p>Here's what I started with.</p> <pre><code>import time x = [] for i in range(100): start = time.clock() for j in range(100000): x.append([]) end = time.clock() print end - start </code></pre> <p>I'm just appending empty lists to the list <code>x</code>. I print out a duration for every 100,000 appends, 100 times. It does slow down like you claimed. (0.03 seconds for the first iteration, and 0.84 seconds for the last... quite a difference.)</p> <p>Obviously, if you instantiate a list but don't append it to <code>x</code>, it runs way faster and doesn't scale up over time.</p> <p>But if you change <code>x.append([])</code> to <code>x.append('hello world')</code>, there's no speed increase at all. The same object is getting added to the list 100 * 100,000 times.</p> <p>What I make of this:</p> <ul> <li>The speed decrease has nothing to do with the size of the list. It has to do with the number of live Python objects.</li> <li>If you don't append the items to the list at all, they just get garbage collected right away and are no longer being managed by Python.</li> <li>If you append the same item over and over, the number of live Python objects isn't increasing. But the list does have to resize itself every once in a while. But this isn't the source of the performance issue.</li> <li>Since you're creating and adding lots of newly created objects to a list, they remain live and are not garbage collected. The slow down probably has something to do with this.</li> </ul> <p>As far as the internals of Python that could explain this, I'm not sure. But I'm pretty sure the list data structure is not the culprit.</p>
    singulars
    1. This table or related slice is empty.
    1. POIs there a way to circumvent Python list.append() becoming progressively slower in a loop as the list grows?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USFogleBird
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIf you are forming a list of 100000 * 100 empty lists at 36 bytes a piece, that's 360 MB of lists you are trying to store on top of the big list and other overhead. Are you sure you aren't running into memory fragmentation or disk swapping?
      singulars
      1. PO
      1. USMike Graham
    2. COFor figuring out which of two similar pieces of code is faster, timing is key (in Python, you'd use the `timeit` module to help you measure.) For determining how things scale, you have to understand mathematically what is going on to characterize the properties of your algorithm; timing can be important but is secondary. One of the most important things when timing an operation is to be sure you control your test to catch exactly what you want.
      singulars
      1. PO
      1. USMike Graham
    3. COIt's not disk swapping or anything. It scales up right away and quite linearly. Not sure what point you're trying to make in your second comment? Feel free to run the examples I posted for yourself to see how it behaves. Do you disagree with my conclusions?
      singulars
      1. PO
      1. USFogleBird
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload