Databases
StackOverflow2013
Postsdbo
PObash script - variable expansion of process substitution
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PObash script - variable expansion of process substitution
    text
    copied!<p>I'm trying to perform analyses on each column of a multi-column file, and using paste, to rejoin the columns together. I don't know a priori how many columns there are, so I use "wc -w" and a loop to define an array of commands. Each command then is a process substitution. The following script shows what I'm trying, and the output is shown after. Notably if I echo the array of commands to the terminal, then, with the mouse, cut-n-paste it, it works fine, so it must be the order of variable expansion and process substitution.</p> <p>In short, I need to have a process substitution <em>inside</em> a shell variable. Any ideas? Thanks in advance.</p> <p>-------------- script.sh ----------------</p> <pre><code>#!/bin/bash f="file.txt"; echo "File contents" cat $f; # simple solution echo; echo "First try"; paste &lt;(cat $f) &lt;(tac $f) # now define cmd[1] and cmd[2] and merge together with paste echo; echo "Second try"; cmd[0]="paste"; cmd[1]="cat $f"; cmd[2]="tac $f"; ${cmd[0]} &lt;(${cmd[1]}) &lt;(${cmd[1]}) # but what I really want is something like: echo; echo "Third try"; cmd[1]="&lt;(cat $f)"; cmd[2]="&lt;(tac $f)"; ${cmd[0]} ${cmd[1]} ${cmd[2]} # or even better: echo; echo "Fourth try"; ${cmd[*]} echo; echo "Show the array"; echo ${cmd[*]} </code></pre> <p>------------- output --------------------</p> <pre><code>$ ./scipt.sh File contents A B C D E F G H I First try A B C G H I D E F D E F G H I A B C Second try A B C A B C D E F D E F G H I G H I Third try paste: &lt;(cat: No such file or directory Fourth try paste: &lt;(cat: No such file or directory Show the array paste &lt;(cat file.txt) &lt;(tac file.txt) $ paste &lt;(cat file.txt) &lt;(tac file.txt) A B C G H I D E F D E F G H I A B C $ </code></pre> <p>In reply to shellter, here is some sample input.</p> <pre><code> 7.74336e-08 7.30689e-08 0.359106 19.981796 -0.160611 0.027 7.74336e-08 7.30689e-08 0.363938 19.985069 0.041319 0.035 7.74336e-08 7.30689e-08 0.363133 19.982094 0.041319 0.068 7.74336e-08 7.30689e-08 0.360716 19.981796 -0.160611 0.006 7.74336e-08 7.30689e-08 0.361522 19.981796 0.243249 0.049 7.74336e-08 7.30689e-08 0.357897 19.986260 0.041319 0.035 </code></pre> <p>There might be 100 million lines of data like this. I need to separate off each column, separate each column into blocks of (say) 1000, then perform an average of each element in the blocks, then merge the averaged columns back together again. For the example below, if I were averaging over just 2 blocks of 3 elements each (instead of 100K of 1000 each), then the output from column 6 would be:</p> <pre><code>0.0165 # =(0.027+0.006)/2 - 1st row from each size-3 block 0.042 # =(0.035+0.049)/2 - 2nd row 0.0515 # =(0.068+0.035)/2 - 3rd row </code></pre> <p>I already have the program to do this averaging (that's "Some_Complicated_Analysis") and it works fine. So all I need to for my script to separate off the columns, feed it into Some_Comp_Analysis, then merge the various outputs back into columns again with <code>paste</code>. But, the files are v. large, and I don't know a priori how many columns there are. If I knew there would be only 2 columns, then <code>paste &lt;(${cmd[1]}) &lt;(${cmd[2]})</code> would work fine.</p> <p><em><strong>SOLUTION FOUND</em></strong></p> <p>UPDATE: an answer has been found - it is as shown in the reply update by glenn jackman below. The <code>paste</code> command must be preceded by <code>eval</code>. I don't know exactly why this is necessary, but without them, the variable expansion <code>${cmd[]}</code> messes up the process substitution <code>&lt;(...)</code>. The answer above also puts double-quotes around the array expansion <code>"${cmd[*]}"</code> however these seem not so important - though without them some other expansions within <code>cmd[]</code> might fail. However, the <code>eval</code> is necessary.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload