Databases
StackOverflow2013
Postsdbo
POIterating csvDictReader with nested for loops
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIterating csvDictReader with nested for loops
    primarykey
    data
    text
    <p>I am having difficulty understanding how to properly iterate over a csvDictReader object. I am trying to return each csv value tied to a specific user-defined datacode. Since I am generating the dict from a web query I would like to re-use it in memory rather than poll the web again for data that is already in memory.</p> <pre><code>import sys, csv, urllib2 class SmfImpl(): def __init__( self, ctx ): self.ctx = ctx self.csv_dict = [] self.flag = ['Not Available', ''] self.ticker = 'XOM' def getMorningKey(self, datacode): return fetch_keyratios(self, datacode) #these functions are not in the SmfImpl class because they're from a seperate file def query_morningstar(self, url_ending): url = 'http://financials.morningstar.com/ajax/exportKR2CSV.html?&amp;callback=?&amp;t=XNYS:%s%s' % (self.ticker, url_ending) req = urllib2.Request(url) response = sniff_query(req) response.readline() return csv.DictReader(response) def sniff_query(req): try: response = urllib2.urlopen(req) except urllib2.URLError: return 'Check Connection' sniff = response.readline() if str(sniff) == '': return 'Not Available' return response def fetch_keyratios(self, datacode): if datacode &lt; 1 or datacode &gt; 990: return 'Invalid Datacode' #check if we already have the data we need if self.flag[0] == 'Check Connection' or self.flag[0] == 'Not Available' or self.flag[1] != self.ticker: #query remote and check for errors self.csv_dict = query_morningstar(self,'&amp;region=usa&amp;culture=en-US&amp;cur=USD&amp;order=desc') if self.csv_dict == 'Check Connection' or self.csv_dict == 'Not Available': self.flag[1] = '' return self.csv_dict else: self.flag[0] = '' self.flag[1] = self.ticker return sort_keyratios(self, datacode) def sort_keyratios(self, datacode): counter = 1 skipped = 0 skip_lines = [15, 16, 26, 36, 37, 57, 58, 64, 65, 86, 91, 92] #iterate through returned dict line by line for line in self.csv_dict: for item in skip_lines: if counter == item: skipped += 1 for val in range(1, len(line)): #match year values to datacodes if datacode == val: return self.csv_dict.fieldnames[val] #match data values to datacodes if datacode-((counter-skipped)*(len(line)-1)) == val: return line[self.csv_dict.fieldnames[val]] counter += 1 return 'No Data' if __name__ == "__main__": smf = SmfImpl(sys.argv) ticker = 'XOM' for val in range (1,24): print ticker, val,':', smf.getMorningKey(val) </code></pre> <p>The original csv is called for by the script, but it can also be found <a href="http://financials.morningstar.com/ratios/r.html?t=XOM&amp;region=USA&amp;culture=en-US" rel="nofollow">here</a></p> <p>The output that I'm getting is:</p> <pre><code>XOM 1 : TTM XOM 2 : 2012-12 XOM 3 : 2011-12 XOM 4 : 2010-12 XOM 5 : 2009-12 XOM 6 : 2008-12 XOM 7 : 2007-12 XOM 8 : 2006-12 XOM 9 : 2005-12 XOM 10 : 2004-12 XOM 11 : 2003-12 XOM 12 : -35,300 XOM 13 : 21,899 XOM 14 : 5.00 XOM 15 : -3,649 XOM 16 : None XOM 17 : 2008-12 XOM 18 : 100.00 XOM 19 : 56.47 XOM 20 : 42.54 XOM 21 : 4.65 XOM 22 : XOM 23 : 13.67 </code></pre> <p>What I am trying to get:</p> <pre><code>XOM 1 : TTM XOM 2 : 2012-12 XOM 3 : 2011-12 XOM 4 : 2010-12 XOM 5 : 2009-12 XOM 6 : 2008-12 XOM 7 : 2007-12 XOM 8 : 2006-12 XOM 9 : 2005-12 XOM 10 : 2004-12 XOM 11 : 2003-12 XOM 12 : 443,708 XOM 13 : 482,295 XOM 14 : 486,429 XOM 15 : 383,221 XOM 16 : 310,586 XOM 17 : 477,359 XOM 18 : 404,552 XOM 19 : 377,635 XOM 20 : 370,680 XOM 21 : 298,035 XOM 22 : 246,738 XOM 23 : 27.8 </code></pre> <p>EDIT: I am trying to map the datacodes in groups per line of the original csv. e.g Years would be datacodes 1 to 11 (TTM to 2003-12), Revenue would be datacodes 12 to 22 (443708 to 246738) etc. Ultimately these datacodes will be moved to user input, so they may be accessed in any order.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USMartijn Pieters
    1. USdman
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COWill the vals/datacodes always be accessed in ascending order like that? Would it be feasible to read the entire csv into memory?
      singulars
      1. POIterating csvDictReader with nested for loops
      1. USmartineau
    2. CONo, both ticker and val that are defined in main will ultimately be moved to user input / sys.argv, so the values may be accessed in any order.
      singulars
      1. POIterating csvDictReader with nested for loops
      1. USdman
    3. COWell, for one thing, that would mean re-reading at least some portion of the csv data each time because `csv.DictReader` accesses the data in a sequential manor -- which was why I suggested reading the whole thing into memory which would allow random access (as well as retaining it between calls). Also your question needs to have additional information added to it explaining the mapping of datacodes to locations in the csv file because it's not clear enough from the code and handful of comments in it. Frankly it's not clear to me why you're even using `csv.DictReader` to try to do this.
      singulars
      1. POIterating csvDictReader with nested for loops
      1. USmartineau
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload