StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POMore efficient way to retrieve first occurrence of every unique value from a csv column in Python
text
Body
copied!<p>A large csv I was given has a large table of flight data. A function I wrote to help parse it iterates over the column of Flight ID's, and then returns a dictionary containing the index and value of every unique Flight ID in order of first appearance.</p> <p>Dictionary = { Index: FID, ... }</p> <p>This comes as a quick adjustment to an older function that didn't require having to worry about FID repeats in the column (a few hundred thousand rows later...).</p> <p>Right now, I have it iterating over and comparing each value in order. If a value is equal to the value after it, it skips it. If the next value is different, it stores the value in the dictionary. I changed it to now also check if that value has already occured before, and if so, to skip it.<br> Here's my code:</p> <pre><code>def DiscoverEarliestIndex(self, number): finaldata = {} columnvalues = self.column(number) columnenum = {} for a, b in enumerate(columnvalues): columnenum[a] = b i = 0 while i < (len(columnvalues) - 1): next = columnenum[i+1] if columnvalues[i] == next: i += 1 else: if next in finaldata.values(): i += 1 continue else: finaldata[i+1]= next i += 1 else: return finaldata </code></pre> <p>It's very inefficient, and slows down as the dictionary grows. The column has 5.2 million rows, so it's obviously not a good idea to handle this much with Python, but I'm stuck with it for now.</p> <p>Is there a more efficient way to write this function?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload