StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POusing generators and cStringIO in python to stream strings
text
Body
copied!<p>I'm trying to read a very large string stream using cStringIO in a python dictionary:</p> <pre><code>def stream_read(self, path): try: # create a string stream from the contents at 'path' # note: the string at self._my_dict[path] is 7MB in size stream = StringIO.StringIO(self._my_dict[path]) while True: # buffer size is 128kB, or 128 * 1024 buf = stream.read(self.buffer_size) if buf != '': yield buf else: raise StopIteration except KeyError: raise IOError("Could not get content") </code></pre> <p>And in my test suite, I'm testing this function by first testing stream_write, asserting that the data exists at that path, and then calling stream_read:</p> <pre><code>def test_stream(self): filename = self.gen_random_string() # test 7MB content = self.gen_random_string(7 * 1024 * 1024) # test stream write io = StringIO.StringIO(content) self._storage.stream_write(filename, io) io.close() self.assertTrue(self._storage.exists(filename)) # test read / write data = '' for buf in self._storage.stream_read(filename): data += buf self.assertEqual(content, data) </code></pre> <p>Yet in my test suite, I'm catching an AssertionError:</p> <pre><code>====================================================================== FAIL: test_stream (test_swift_storage.TestSwiftStorage) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/bacongobbler/.../test/test_local_storage.py", line 44, in test_stream self.assertEqual(content, data) AssertionError: '[squelched]' != '<cStringIO.StringI object at 0x3148e70>' ---------------------------------------------------------------------- Ran 28 tests in 20.495s FAILED (failures=1) </code></pre> <p>It looks related to <a href="https://stackoverflow.com/questions/20429971/issues-working-with-python-generators-and-openstack-swift-client">an issue I posted last week</a>, but I'm still not quite sure I understand why <code>stream</code> is getting set to the Generator as a string in this case.</p> <p>If anyone wants to take a closer look at the source code, it's all up at <a href="https://github.com/bacongobbler/docker-registry/blob/106-swift-storage/test/utils/mock_swift_storage.py" rel="nofollow noreferrer">https://github.com/bacongobbler/docker-registry/blob/106-swift-storage/test/utils/mock_swift_storage.py</a></p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload