Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>From the docs:</p> <p><a href="http://docs.python.org/2.7/library/logging.handlers.html#watchedfilehandler" rel="nofollow">http://docs.python.org/2.7/library/logging.handlers.html#watchedfilehandler</a></p> <blockquote> <p>The WatchedFileHandler class, located in the logging.handlers module, is a FileHandler which watches the file it is logging to. If the file changes, it is closed and reopened using the file name.</p> </blockquote> <p>The WatchedFileHandler works on Linux, but not on Windows (because it does not have inodes). It monitors changes on the file inode. If anything is changed on that file, it is closed and opened again. </p> <p>Depending on how the WatchedFileHandler is implemented, I think they repeat this process until it can safely write to the file, but that would probably mean that your processes are in constant open/close battle for the log file.</p> <p>Correction: it tries once to close and then it reopens the file again. I'm looking for another explanation about why you don't have problems with it.</p> <p>So far, I did not find any reason to explain why you don't have intermixed lines in your logs. The WatchedFileHandler was designed to be used by one application (controlling multithread access using a RLock before emit, with a valuable exception to support external changes like log rotation). But looking the source code of its super classes, I did not find any reason. An unchecked theory, is that:</p> <p>a) As the log is always opened in append mode</p> <p>b) As it checks once, before writing, if anything changed</p> <p>c) You always get a complete line because your lines are smaller than the writing buffer. So you start from the end of the last line, write a new line and the process repeats again and again. If two process write at the same time, they will start to write their new lines from the end of the last line.</p> <p>Try to look for anomalies on the log lines, like missing lines... if the size of the log is very regular, it can be difficult to check a concurrency problem. I would look for anomalies on the last line of the log too.</p> <p>How many processors your system has?</p> <p>Test program to break the log:</p> <pre><code>import logging import logging.handlers import sys FORMAT = "%(asctime)-15s %(process)5d %(message)s" logger = logging.getLogger(__name__) wfh = logging.handlers.WatchedFileHandler("test.log", "a") wfh.setFormatter(logging.Formatter(FORMAT)) logger.addHandler(wfh) logger.setLevel(logging.DEBUG) n = int(sys.argv[1]) for x in range(1000): logger.info("%6d %s" % (x, str(n)*(4096+n))) </code></pre> <p>Run with a simple shell script:</p> <pre><code>python test.py 1 &amp; python test.py 2 &amp; python test.py 3 &amp; python test.py 4 &amp; python test.py 5 &amp; python test.py 6 &amp; python test.py 7 &amp; python test.py 8 &amp; python test.py 9 &amp; </code></pre> <p>Run on a CentOS 6.3 with 4 processors, I start to see anomalies on the lines. You can change the size of the line in the program, small lines look to be ok, but large lines are not.</p> <p>You can test with:</p> <pre><code> grep -v ^2013 test.log </code></pre> <p>If the log is ok, all lines would start with 2013. The grep will not list mixed lines, but I could find them easily with less. </p> <p>So, it works for you because your lines are smaller than your writing buffer or because your processes don't write to the log very often, but even them, there is no guarantee that it will works all the time.</p>
    singulars
    1. This table or related slice is empty.
    1. POWhy don’t I see interleaved lines when writing to a file in parallel?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USnmenezes
    1. USnmenezes
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POWhy don’t I see interleaved lines when writing to a file in parallel?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COI’m pretty sure that this refers to the changes in file metadata, specifically the inode, which happens on rotation. It would be very strange to reopen the log file after every write; nor does it guarantee atomicity. I’m afraid your answer is not helpful without additional references.
      singulars
      1. PO
      1. USVasiliy Faronov
    2. COIn fact, you can see it in the source code: [`Lib/logging/handlers.py:462`](http://hg.python.org/cpython/file/tip/Lib/logging/handlers.py#l462).
      singulars
      1. PO
      1. USVasiliy Faronov
    3. COI think your append mode reference is a step in the right direction. See discussion of `O_APPEND` here: http://stackoverflow.com/questions/10650861/atomicity-of-write2-to-a-local-filesystem
      singulars
      1. PO
      1. USVasiliy Faronov
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload