Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p><code>READAHEAD_BUFSIZE</code> is <em>only</em> used when you use the file as an iterator:</p> <pre><code>for line in fileobj: print line </code></pre> <p>It is a separate buffer from the normal buffer argument, which is handled by the <code>fread</code> C API calls. Both are used when iterating.</p> <p>From <a href="http://docs.python.org/2/library/stdtypes.html#file.next" rel="nofollow"><code>file.next()</code></a>:</p> <blockquote> <p>In order to make a <code>for</code> loop the most efficient way of looping over the lines of a file (a very common operation), the <code>next()</code> method uses a hidden read-ahead buffer. As a consequence of using a read-ahead buffer, combining <code>next()</code> with other file methods (like <code>readline()</code>) does not work right. However, using <code>seek()</code> to reposition the file to an absolute position will flush the read-ahead buffer.</p> </blockquote> <p>The OS buffer size is not changed, the <code>setvbuf</code> is done when the file is opened and not touched by the file iteration code. Instead, calls to <code>Py_UniversalNewlineFread</code> (which uses <code>fread</code>) are used to fill the read-ahead buffer, creating a <em>second</em> buffer internal to Python. Python otherwise leaves the regular buffering up to the C API calls (<code>fread()</code> calls are buffered; the userspace buffer is consulted by <code>fread()</code> to satisfy the request, Python doesn't have to do anything about that).</p> <p><code>readahead_get_line_skip()</code> then serves lines (newline terminated) from this buffer. If the buffer no longer contains newlines, it'll refill the buffer by recursing over itself with a buffer size 1.25 times the previous value. This means that file iteration can read the whole rest of the file into the memory buffer if there are no more newline characters in the whole file!</p> <p>To see how much the buffer reads, print the file position (using <code>fileobj.tell()</code>) while looping:</p> <pre><code>&gt;&gt;&gt; with open('test.txt') as f: ... for line in f: ... print f.tell() ... 8192 # 1 times the buffer size 8192 8192 ~ lines elided 18432 # + 1.25 times the buffer size 18432 18432 ~ lines elided 26624 # + 1 times the buffer size; the last newline must've aligned on the buffer boundary 26624 26624 ~ lines elided 36864 # + 1.25 times the buffer size 36864 36864 </code></pre> <p>etc.</p> <p>What bytes are actually read from the disk (provided <code>fileobj</code> is an actual physical file on your disk) depend not only on the interplay between the <code>fread()</code> buffer and the internal read-ahead buffer; but also if the OS itself is using buffering. It could well be that even if the file buffer is exhausted, the OS serves the system call to read from the file from it's own cache instead of going to the physical disk.</p>
    singulars
    1. This table or related slice is empty.
    1. POWhat is the difference between the buffering argument to open() and the hardcoded readahead buffer size used when iterating through a file?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USMartijn Pieters
    1. USMartijn Pieters
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POWhat is the difference between the buffering argument to open() and the hardcoded readahead buffer size used when iterating through a file?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COBut `setvbuf` is called when you call `open`. Presumably somewhere inside `file_iternext` an OS-level read is called, so this read overwrites the buffer size set with `setvbuf` in order to use `READAHEAD_BUFSIZE`?
      singulars
      1. PO
      1. USEmily
    2. CONo, it reads data into an *internal* buffer.
      singulars
      1. PO
      1. USMartijn Pieters
    3. COSorry, but I still don't understand. I'm looking at two functions, `file_iternext` and `file_read`. Both get data from the file by calling `Py_UniversalNewlineFread`, passing it `bufsize` equal to `READAHEAD_BUFSIZE` or the size of the file, respectively. `Py_UniversalNewlineFread` then reads `bufsize` bytes by calling `fread`. Maybe `fread` doesn't use the buffer set by `setvbuf`, but then I'm confused as to what the point of the `buffering` argument is: what file operations does it set the buffer size for?
      singulars
      1. PO
      1. USEmily
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload