Databases
StackOverflow2013
Postsdbo
PObackground jobs' garbled output
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PObackground jobs' garbled output
    primarykey
    data
    text
    <p>I'm trying to gather line by line the output of several child processes in a bash script, in order to forward it to another process.</p> <p>I found nothing that guarantees that the outputs of the sub-processes are not mixed, but it's important to me that each output line goes properly to the output. The order between the outputs does not matters.</p> <p>Here's an example of mixed/garbled output:</p> <pre><code>#!/bin/bash for i in {1..1000}; do ( { echo BEGIN; dmesg; echo END; } | tr -d '\n'; echo ) &amp; done wait </code></pre> <p>Running this:</p> <pre><code>$ ./test_output.sh | perl -ne 'print "$1\n" if m/(.{1,20}BEGIN.{0,20})/' | head 0.000000] SRAT: PXMBEGIN[ 0.000000] Initi ME through PCIe PME BEGIN[ 0.000000] Initi ME through PCIe PME BEGIN[ 0.000000] Initi [ 0.209816] pci 0BEGIN[ 0.000000] Initi ciehp 0000:00:16.1:pBEGIN[ 0.000000] Initi CI: Updating contextBEGIN[ 0.000000] Initi l family 2[ 0.588BEGIN[ 0.000000] Initi ME through PCIe PME BEGIN[ 0.000000] Initi CI: Updating contextBEGIN[ 0.000000] Initi 3922 pages, LIFO batBEGIN[ 0.000000] Initi </code></pre> <p>You can see several line with mixed content.</p> <p>Of course, without the <code>&amp;</code> everything is fine.</p> <p>So for now, I have no choice but to redirect every child's output to a file and then after a big <code>wait</code>, <code>cat</code> all these files.</p> <p>Running the same with GNU parallel does part of the job, but it's not an option in my environment.</p> <blockquote> <p>GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU parallel as input for other programs.</p> </blockquote> <p>So GNU parallel will write each job output as soon as each job is finished and it takes care of not mixing the output. That's good. But I would be also interested in getting the output of each job as soon as possible, ie not waiting for the job exit. There's the "-u" switch, but it will then mix the jobs output.</p> <p>Do I need to play with fifo, select, or even write a perl script ?</p> <p>--</p> <p>I think i've found why/how/when the outputs get mixed in man 7 pipe</p> <blockquote> <p>POSIX.1-2001 says that write(2)s of less than PIPE_BUF bytes must be atomic: the output data is written to the pipe as a contiguous sequence. Writes of more than PIPE_BUF bytes may be nonatomic: the kernel may interleave the data with data written by other processes. POSIX.1-2001 requires PIPE_BUF to be at least 512 bytes. (On Linux, PIPE_BUF is 4096 bytes.)</p> </blockquote>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USBenoît
    1. USBenoît
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PObackground jobs' garbled output
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PObackground jobs' garbled output
      1. USAdrian Frühwirth
      1. VTFavorite
    3. VO
      singulars
      1. PObackground jobs' garbled output
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIs it important that you get any output as soon as possible, or is it fine if you get it all once everything is finished?
      singulars
      1. PObackground jobs' garbled output
      1. USTeddy
    2. COI'd like to get it as soon as possible, yes. Otherwise, whether you store it in memory or a file per process, I think that's the same solution.
      singulars
      1. PObackground jobs' garbled output
      1. USBenoît
    3. COCan you elaborate why a 10 seconds install of GNU Parallel (wget -O - pi.dk/3 | sh) will not work? Is it covered by: http://oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html
      singulars
      1. PObackground jobs' garbled output
      1. USOle Tange
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload