Databases
StackOverflow2013
Postsdbo
POhow to use awk and a conditional pipe to submit qsub jobs?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POhow to use awk and a conditional pipe to submit qsub jobs?
    primarykey
    data
    text
    <p>I have a file (fasta) that I am using awk to extract the needed fields from (sequences with their headers). I then pipe it to a BLAST program and finally I pipe it to qsub in order to submit a job. the file:</p> <pre><code>&gt;sequence_1 ACTGACTGACTGACTG &gt;sequence_2 ACTGGTCAGTCAGTAA &gt;sequence_3 CCGTTGAGTAGAAGAA </code></pre> <p>and the command (which works):</p> <pre><code>awk &lt; fasta.fasta '/^&gt;/ { print $0 } $0 !~ /^&gt;/' | echo "/Local/ncbi-blast-2.2.25+/bin/blastx -db blastdb.fa -outfmt 5 &gt;&gt; /User/blastresult.xml" | qsun -q S </code></pre> <p>what I would like to do is a add a condition that will sample the number of jobs I am running (using qstat) if it is below a certain threshold the job will be submitted. for example:</p> <pre><code>allowed_jobs=200 #for example awk &lt; fasta.fasta '/^&gt;/ { print $0 } $0 !~ /^&gt;/' | echo "/Local/ncbi-blast-2.2.25+/bin/blastx -db blastdb.fa -outfmt 5 &gt;&gt; /User/blastresult.xml" | cmd=$(qstat -u User | grep -c ".") | if [ $cmd -lt $allowed_jobs ]; then qsub -q S </code></pre> <p>unfortunately (for me anyway) I have failed in all my attempts to do that. I'd be grateful for any help</p> <p>EDIT: elaborating a bit: what I am trying to do is to extract from the fasta file this:</p> <pre><code>&gt;sequene_x ACTATATATATA </code></pre> <p>or basically: >HEADER\nSEQUENCE one by one and pipe it to the blast program which can take stdin. I want to create a unique job for each sequence and this is the reason I want to pipe to qsub for each sequence. to put it plainly the qsub submission would have looked something like this:</p> <pre><code>qsub -q S /Local/ncbi-blast-2.2.25+/bin/blastx -db blastdb.fa -query FASTA_SEQUENCE -outfmt 5 &gt;&gt; /User/blastresult.xml </code></pre> <p>note that the -query flag is unnecessary if stdin sequence is piped to it. however, the main problem for me is how to incorporate the condition I mentioned above so that the sequence will be piped to qsub only if the qstat result is below a threshold. ideally if the qstat result is above the threshold it'll sleep until i goes below and then pass it forward.</p> <p>thanks.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USSchrodinger's Cat
    1. USSchrodinger's Cat
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COIf `qsub` runs in the background, can't you just count the number of processes with `ps ax | grep -wc '[q]sub'`? Or, better than just not running jobs, why not submit them to a queue, then write a queue runner that spawns `qsub` on the first 200 jobs, then spawns new instances as the old ones finish? That would be an entirely different question, of course.
      singulars
      1. POhow to use awk and a conditional pipe to submit qsub jobs?
      1. USghoti
    2. COYour first awk call does not do filtering: it prints the whole file, the same way cat does. In fact, the output from your awk command is not used at all: the echo command does not take input from the standard input (the pipe in this case). Perhaps, you tell us what you try to accomplish: what the input looks like, what the output should be.
      singulars
      1. POhow to use awk and a conditional pipe to submit qsub jobs?
      1. USHai Vu
    3. CO@HaiVu I have added some more information about my problem above. thanks.
      singulars
      1. POhow to use awk and a conditional pipe to submit qsub jobs?
      1. USSchrodinger's Cat
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload