Databases
StackOverflow2013
Postsdbo
POHow to use a python context manager inside a generator
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to use a python context manager inside a generator
    primarykey
    data
    text
    <p>In python, should with-statements be used inside a generator? To be clear, I am not asking about using a decorator to create a context manager from a generator function. I am asking whether there is an inherent issue using a with-statement as a context manager inside a generator as it will catch <code>StopIteration</code> and <code>GeneratorExit</code> exceptions in at least some cases. Two examples follow.</p> <p>A good example of the issue is raised by Beazley's example (page 106). I have modified it to use a with statement so that the files are explicitly closed after the yield in the opener method. I have also added two ways that an exception can be thrown while iterating the results.</p> <pre><code>import os import fnmatch def find_files(topdir, pattern): for path, dirname, filelist in os.walk(topdir): for name in filelist: if fnmatch.fnmatch(name, pattern): yield os.path.join(path,name) def opener(filenames): f = None for name in filenames: print "F before open: '%s'" % f #f = open(name,'r') with open(name,'r') as f: print "Fname: %s, F#: %d" % (name, f.fileno()) yield f print "F after yield: '%s'" % f def cat(filelist): for i,f in enumerate(filelist): if i ==20: # Cause and exception f.write('foobar') for line in f: yield line def grep(pattern,lines): for line in lines: if pattern in line: yield line pylogs = find_files("/var/log","*.log*") files = opener(pylogs) lines = cat(files) pylines = grep("python", lines) i = 0 for line in pylines: i +=1 if i == 10: raise RuntimeError("You're hosed!") print 'Counted %d lines\n' % i </code></pre> <p>In this example, the context manager successfully closes the files in the opener function. When an exception is raised, I see the trace back from the exception, but the generator stops silently. If the with-statement catches the exception why doesn't the generator continue?</p> <p>When I define my own context managers for use inside a generator. I get runtime errors saying that I have ignored a <code>GeneratorExit</code>. For example:</p> <pre><code>class CManager(object): def __enter__(self): print " __enter__" return self def __exit__(self, exctype, value, tb): print " __exit__; excptype: '%s'; value: '%s'" % (exctype, value) return True def foo(n): for i in xrange(n): with CManager() as cman: cman.val = i yield cman # Case1 for item in foo(10): print 'Pass - val: %d' % item.val # Case2 for item in foo(10): print 'Fail - val: %d' % item.val item.not_an_attribute </code></pre> <p>This little demo works fine in case1 with no exceptions raised, but fails in case2 where an attribute error is raised. Here I see a <code>RuntimeException</code> raised because the with statement has caught and ignored a <code>GeneratorExit</code> exception.</p> <p>Can someone help clarify the rules for this tricky use case? I suspect it is something I am doing, or not doing in my <code>__exit__</code> method. I tried adding code to re-raise <code>GeneratorExit</code>, but that did not help.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJohn La Rooy
    1. USDavid
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow to use a python context manager inside a generator
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow to use a python context manager inside a generator
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POHow to use a python context manager inside a generator
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload