Databases
StackOverflow2013
Postsdbo
POPossible race condition with piped output from multiple tee recipients arriving out-of-sequence on a named pipe in a BASH script
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPossible race condition with piped output from multiple tee recipients arriving out-of-sequence on a named pipe in a BASH script
    primarykey
    data
    text
    <p>UPDATE: While not actually solving the original problem presented with regards to my piping endeavours, I've solved my problem by simplifying it greatly, and just ditching the pipes altogether. Here's a proof-of-concept script that generates, in parallel while reading just once from disk, CRC32, MD5, SHA1, SHA224, SHA256, SHA384, and SHA512 checksums, and returning them as a JSON object (gonna use the output in PHP and Ruby). It's crude without error checking, but it works:</p> <pre><code>#!/bin/bash checksums="`tee &lt;"$1" \ &gt;( cfv -C -q -t sfv -f - - | tail -n 1 | sed -e 's/^.* \([a-fA-F0-9]\{8\}\)$/"crc32":"\1"/' ) \ &gt;( md5sum - | sed -e 's/^\([a-fA-F0-9]\{32\}\) .*$/"md5":"\1"/' ) \ &gt;( sha1sum - | sed -e 's/^\([a-fA-F0-9]\{40\}\) .*$/"sha1":"\1"/' ) \ &gt;( sha224sum - | sed -e 's/^\([a-fA-F0-9]\{56\}\) .*$/"sha224":"\1"/' ) \ &gt;( sha256sum - | sed -e 's/^\([a-fA-F0-9]\{64\}\) .*$/"sha256":"\1"/' ) \ &gt;( sha384sum - | sed -e 's/^\([a-fA-F0-9]\{96\}\) .*$/"sha384":"\1"/' ) \ &gt;( sha512sum - | sed -e 's/^\([a-fA-F0-9]\{128\}\) .*$/"sha512":"\1"/') \ &gt;/dev/null`\ " json="{" for checksum in $checksums; do json="$json$checksum,"; done echo "${json:0: -1}}" </code></pre> <p>THE ORIGINAL QUESTION:</p> <p>I'm a bit afraid to ask this question, as I got so many hits on my search phrase that after applying the knowledge harvested from <a href="https://stackoverflow.com/questions/4290684/using-named-pipes-with-bash-problem-with-data-loss">Using named pipes with bash - Problem with data loss</a>, and reading through another 20 pages, I still am at a bit of a stand-still with this.</p> <p>So, to continue nevertheless, I'm doing a simple script to enable me to concurrently create CRC32, MD5, and SHA1 checksums on a file while only reading it from disk once. I'm using cfv for that purpose.</p> <p>Originally, I just hacked together a simply script that wrote cat'ted the file to tee with three cfv commands writing to three separate files under /tmp/, and then attempted to cat them out to stdout afterwards, but ended up with empty output unless I made my script sleep for a second before trying to read the files. Thinking that was weird, I assumed I was a moron in my scripting, so I tried to do a different approach by having the cfv workers output to a named pipe instead. So far, this is my script, after having applied techniques from forementioned link:</p> <pre><code>!/bin/bash # Bail out if argument isn't a file: [ ! -f "$1" ] &amp;&amp; echo "'$1' is not a file!" &amp;&amp; exit 1 # Choose a name for a pipe to stuff with CFV output: pipe="/tmp/pipe.chksms" # Don't leave an orphaned pipe on exiting or being terminated: trap "rm -f $pipe; exit" EXIT TERM # Create the pipe (except if it already exists (e.g. SIGKILL'ed b4)): [ -p "$pipe" ] || mkfifo $pipe # Start a background process that reads from the pipe and echoes what it # receives to stdout (notice the pipe is attached last, at done): while true; do while read line; do [ "$line" = "EOP" ] &amp;&amp; echo "quitting now" &amp;&amp; exit 0 echo "$line" done done &lt;$pipe 3&gt;$pipe &amp; # This 3&gt; business is to make sure there's always # at least one producer attached to the pipe (the # consumer loop itself) until we're done. # This sort of works without "hacks", but tail errors out when the pipe is # killed, naturally, and script seems to "hang" until I press enter after, # which I believe is actually EOF to tail, so it's no solution anyway: #tail -f $pipe &amp; tee &lt;"$1" &gt;( cfv -C -t sfv -f - - &gt;$pipe ) &gt;( cfv -C -t sha1 -f - - &gt;$pipe ) &gt;( cfv -C -t md5 -f - - &gt;$pipe ) &gt;/dev/null #sleep 1s echo "EOP" &gt;$pipe exit </code></pre> <p>So, executed as it stands, I get this output:</p> <pre><code>daniel@lnxsrv:~/tisso$ ./multisfv file : : : quitting now - : Broken pipe (CF) close failed in file object destructor: sys.excepthook is missing lost sys.stderr - : Broken pipe (CF) close failed in file object destructor: sys.excepthook is missing lost sys.stderr - : Broken pipe (CF) daniel@lnxsrv:~/tisso$ close failed in file object destructor: sys.excepthook is missing lost sys.stderr </code></pre> <p>But, with the sleep 1s commented out, I the get expected output, </p> <pre><code>daniel@lnxsrv:~/tisso$ ./multisfv file 3bc1b5ff125e03fb35491e7d67014a3e * -: 1 files, 1 OK. 0.013 seconds, 79311.7K/s 5e3bb0e3ec410a8d8e14fef1a6daababfc48c7ce * -: 1 files, 1 OK. 0.016 seconds, 62455.0K/s ; Generated by cfv v1.18.3 on 2012-03-09 at 23:45.23 ; 2a0feb38 -: 1 files, 1 OK. 0.051 seconds, 20012.9K/s quitting now </code></pre> <p>This puzzles me, as I'd assume that tee wouldn't exit until after each cfv recipient it forks data to has exited, and thus the echo "EOP" statement would execute until all cfv substreams have finished, which would mean they'd have written their output to my named pipe... And then the echo statement would execute.</p> <p>As the behavior is the same without pipes, just using output temporary files, I'm thinking this must be some race condition having to do with the way tee pushes data onto its recipients? I tried a simple "wait" command, but it'll of course wait for my bash child process - the while loop - to finish, so I just get a hanging process.</p> <p>Any ideas?</p> <p>TIA, Daniel :)</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USDanielSmedegaardBuus
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPossible race condition with piped output from multiple tee recipients arriving out-of-sequence on a named pipe in a BASH script
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPossible race condition with piped output from multiple tee recipients arriving out-of-sequence on a named pipe in a BASH script
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POPossible race condition with piped output from multiple tee recipients arriving out-of-sequence on a named pipe in a BASH script
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI would expect there is source code available for these checksums. How about combining them into 1 program, and writing out 3 values that you process into the appropriate checksum file. I got to believe perl probably has modules for this, which again, you could glom together to just do 1 pass on the file. (Just thinking out side the box on this, YRMV). Good luck!
      singulars
      1. POPossible race condition with piped output from multiple tee recipients arriving out-of-sequence on a named pipe in a BASH script
      1. USshellter
    2. COWould this help? `parallel --group 'cfv -C -t sfv -f {} - ;cfv -C -t sha1 -f {} - ;cfv -C -t md5 -f {} - ;' ::: file`
      singulars
      1. POPossible race condition with piped output from multiple tee recipients arriving out-of-sequence on a named pipe in a BASH script
      1. USpotong
    3. CO@shelter - I guess writing my own routines is always my fallback, but I'd prefer to use tools already available as much as it's possible.
      singulars
      1. POPossible race condition with piped output from multiple tee recipients arriving out-of-sequence on a named pipe in a BASH script
      1. USDanielSmedegaardBuus
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload