Databases
StackOverflow2013
Postsdbo
POusing awk to grep from each bunch of multiple lines
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POusing awk to grep from each bunch of multiple lines
    primarykey
    data
    text
    <p>I have a log file which has bunch of lines, where each bunch is separated by blank lines. I want to grep certain lines (containing common pattern) from each bunch of line. Each bunch of line is about a mail. A sample log file is as follows:</p> <pre class="lang-none prettyprint-override"><code>#START# 03:48:19:798: : &lt;23/08/2012 03:48:19:019&gt; 03:48:19:798: : &lt;---23/08/2012 03:48 ---&gt; 03:48:19:799: : MAIL FROM IP=1.2.3.4 03:48:19:799: : START CHECKING OF IPLIMIT 03:48:19:799: : STOP CHECKING OF IPLIMIT 03:48:20:848:In : MAIL FROM: &lt;a@abc.com&gt; 03:48:20:848: : [A:A:A] 03:48:20:849: : max attach size--&gt;5242880 03:48:20:856: : User Is Authenticated with "a@abc.com and domain abc.com" 03:48:20:856: : Passed 03:48:20:987:In : RCPT TO: &lt;x@xyz.com&gt; 03:48:20:987: : email x@xyz.com 03:48:20:992: : [A:A:A] 03:48:20:999: : passed 03:48:20:999:Inside the Store Mails 03:48:20:999: : BCC feature is not applicable x@xyz.com 03:48:21:000: : BCC feature is not applicable from a@abc.com 03:48:21:000:Inside the Store 03:48:21:132:In : RCPT TO: &lt;y@xyz.com&gt; 03:48:21:132: : email y@xyz.com 03:48:21:133: : [A:A:A] 03:48:21:140: : passed 03:48:21:140:Inside the Store Mails 03:48:21:140: : BCC feature is not applicable y@xyz.com 03:48:21:140: : not authenticated 03:48:21:140:Inside the Store 03:48:21:271: : Data Received 03:50:32:049: : 552 Size Limit Exceeded(5242880) 03:50:32:049: : File Moved in LargeSize Folder.... 03:50:32:049: : File Moved in LargeSize Folder.... 03:50:32:049: : Connection closed 03:50:32:049: : File Deleted /home/Mail//mailbox/LargeSize/x@xyz.com:24085.444724474357(1345673901000) 03:50:32:051: : File Deleted /home/Mail//mailbox/LargeSize/y@xyz.com:39872.512978520455(1345673901140) MAIL DATA : : 6815779 Bytes Total: Conn : 16713 Quit By Host : 5565 Stored : 11134 Loop:0 #END# W A R N I N G ---------------W A R N I N G ...Waiting for activity on port Total Thread Started &amp; 16732 Stoped 16730 #START# 03:56:20:790: : &lt;23/08/2012 03:56:20:020&gt; 03:56:20:790: : &lt;---23/08/2012 03:56 ---&gt; 03:56:20:791: : MAIL FROM IP=2.3.4.5 03:56:20:792: : IP IS FRIEND IN WHITELIST 03:56:20:834:In : MAIL FROM:&lt;y@xyz.com&gt; 03:56:20:834: : [A:A:A] 03:56:20:834: : null 03:56:20:834: : Passed 03:56:20:834:In : RCPT TO: &lt;a@abc.com&gt; 03:56:20:834: : email a@abc.com 03:56:20:835: : Mailing List 03:56:20:835: : [A:A:A] 03:56:20:836: : passed 03:56:20:836: : Proceesing maillist 03:56:20:839: : Data Received 03:56:20:865: : /home/Mail//mailbox/MailingList/a@abc.com:79602.39544573233(1345674380836) Msg Queued For Delivery 03:56:20:865: : Msg forward successfully 03:56:20:865: : /home/Mail//mailbox/MailingList/M14310.39892966699(1345674380837) Msg Queued For Delivery MAIL DATA : : 27985 Bytes Total: Conn : 16732 Quit By Host : 5582 Stored : 11135 Loop:0 #END# ...Waiting for activity on port Total Thread Started &amp; 16735 Stoped 16731 #START# 03:56:23:957: : &lt;23/08/2012 03:56:23:023&gt; 03:56:23:957: : &lt;---23/08/2012 03:56 ---&gt; 03:56:23:958: : MAIL FROM IP=2.3.4.5 03:56:23:959: : IP IS FRIEND IN WHITELIST 03:56:23:999:In : MAIL FROM: &lt;x@xyz.com&gt; 03:56:23:999: : [A:A:A] 03:56:23:999: : null 03:56:23:999: : Passed 03:56:23:999:In : RCPT TO: &lt;y@xyz.com&gt; 03:56:23:999: : email y@xyz.com 03:56:24:000: : [A:A:A] 03:56:24:007: : passed 03:56:24:008:Inside the Store Mails 03:56:24:009: : BCC feature is not applicable y@xyz.com 03:56:24:009: : not authenticated 03:56:24:009:Inside the Store 03:56:24:009: : Data Received 03:56:24:053: : /home/Mail//mailbox/External/y@xyz.com:50098.70335800691(1345674384009) Msg Queued For Delivery 03:56:24:054: : Msg forward successfully MAIL DATA : : 28276 Bytes Total: Conn : 16735 Quit By Host : 5582 Stored : 11136 Loop:0 #END# </code></pre> <p>Here, a@abc.com is an external mail id, and x@xyz.com, y@xyz.com are internal mail ids. For each mail, the bunch of lines starting from #START# to #END# are generated.</p> <p>From each bunch of lines I want to run some pattern matching. I only want those bunch of lines where mail is from an internal email id to external email id (second bunch of line).</p> <p>I don't want bunch of lines where mail is from external email address/id to internal email id (1st bunch of line), or from an internal email id to internal email id. (3rd bunch of line).</p> <p>And after I have the bunch of line where mail is from internal to external, I want to extract the line containing the word <code>FROM</code> and <code>TO</code>.</p> <p>I tried using the <code>RS</code>, <code>ORS</code>, <code>FS</code> and <code>OFS</code> variables of awk to convert each bunch of line starting from and ending at <code>#START#</code> to make a single-line record, but couldn't. I could not replace the newlines by a separator such as <code>|</code> or <code>~</code>. Also, I don't now how to run multiple pattern matching on each resource record. </p> <p>I tried using <code>/PATTERN/</code> option, but then could not run the grep command using <code>system()</code> function to get the lines to check the domain names. it gave me errors: <code>sh: 1: not found</code>. Could not break through it. I used the code:</p> <pre><code>if ($0 ~ /FROM/) { print $0 | system("egrep -i 'FROM|TO'") } </code></pre> <p>Also, if I try to export each record using following type of code, its not working:</p> <pre><code>for i in $(cat log_file | awk_file_givin_1_resource_record_at_a_time) ; do pattern_matching_commands ; done </code></pre> <p>It's no working cause the pattern matching is working on a line at a time, while I want it to work on the entire bunch at a time.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USGautam Somani
    1. USGautam Somani
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POusing awk to grep from each bunch of multiple lines
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POusing awk to grep from each bunch of multiple lines
      1. USatolani
      1. VTFavorite
    1. COThis feels a little too broad; there are many individual questions here. I'd try to break the problem into steps, and solve each step individually. If you have a problem with a particular step, that would make a better, focused question to ask here.
      singulars
      1. POusing awk to grep from each bunch of multiple lines
      1. USchepner
    2. CO@chepner: I think if I manage to take each bunch of line in a variable, and use that variable with bash commands, I may be able to perform normal bash operations on it, though many, to extract the info I want.
      singulars
      1. POusing awk to grep from each bunch of multiple lines
      1. USGautam Somani
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload