StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POCalculate time based metrics(hourly)
text
Body
copied!<p>How would I calculate time-based metrics (hourly average) based on log file data? </p> <p>let me make this more clear, consider a log file that contains entries as follows: Each UIDs appears only twice in the log. they will be in embeded xml format. And they will likely appear OUT of sequence. And the log file will have data for only one day so only one day records will be there.</p> <p>No of UIDs are 2 millions in log file.</p> <p>I have to find out average hourly reponse time for these requests. Below has request and response in log file. UID is the key to relate b/w request and response. </p> <pre><code>2013-04-03 08:54:19,451 INFO [Logger] <?xml version="1.0" encoding="UTF-8" standalone="yes"?><log-event><message-time>2013-04-03T08:54:19.448-04:00</message-time><caller>PCMC.common.manage.springUtil</caller><body>&lt;log-message-body&gt;&lt;headers&gt;&amp;lt;FedDKPLoggingContext id="DKP_DumpDocumentProperties" type="context.generated.FedDKPLoggingContext"&amp;gt;&amp;lt;logFilter&amp;gt;7&amp;lt;/logFilter&amp;gt;&amp;lt;logSeverity&amp;gt;255&amp;lt;/logSeverity&amp;gt;&amp;lt;schemaType&amp;gt;PCMC.MRP.DocumentMetaData&amp;lt;/schemaType&amp;gt;&amp;lt;UID&amp;gt;073104c-4e-4ce-bda-694344ee62&amp;lt;/UID&amp;gt;&amp;lt;consumerSystemId&amp;gt;JTR&amp;lt;/consumerSystemId&amp;gt;&amp;lt;consumerLogin&amp;gt;jbserviceid&amp;lt;/consumerLogin&amp;gt;&amp;lt;logLocation&amp;gt;Beginning of Service&amp;lt;/logLocation&amp;gt;&amp;lt;/fedDKPLoggingContext&amp;gt;&lt;/headers&gt;&lt;payload&gt; &amp;lt;ratedDocument&amp;gt; &amp;lt;objectType&amp;gt;OLB_BBrecords&amp;lt;/objectType&amp;gt; &amp;lt;provider&amp;gt;JET&amp;lt;/provider&amp;gt; &amp;lt;metadata&amp;gt;&amp;amp;lt;BooleanQuery&amp;amp;gt;&amp;amp;lt;Clause occurs=&amp;amp;quot;must&amp;amp;quot;&amp;amp;gt;&amp;amp;lt;TermQuery fieldName=&amp;amp;quot;RegistrationNumber&amp;amp;quot;&amp;amp;gt;44565153050735751&amp;amp;lt;/TermQuery&amp;amp;gt;&amp;amp;lt;/Clause&amp;amp;gt;&amp;amp;lt;/BooleanQuery&amp;amp;gt;&amp;lt;/metadata&amp;gt; &amp;lt;/ratedDocument&amp;gt; &lt;/payload&gt;&lt;/log-message-body&gt;</body></log-event> 2013-04-03 08:54:19,989 INFO [Logger] <?xml version="1.0" encoding="UTF-8" standalone="yes"?><log-event><message-time>2013-04-03T08:54:19.987-04:00</message-time><caller>PCMC.common.manage.springUtil</caller><body>&lt;log-message-body&gt;&lt;headers&gt;&amp;lt;fedDKPLoggingContext id="DKP_DumpDocumentProperties" type="context.generated.FedDKPLoggingContext"&amp;gt;&amp;lt;logFilter&amp;gt;7&amp;lt;/logFilter&amp;gt;&amp;lt;logSeverity&amp;gt;255&amp;lt;/logSeverity&amp;gt;&amp;lt;schemaType&amp;gt;PCMC.MRP.DocumentMetaData&amp;lt;/schemaType&amp;gt;&amp;lt;UID&amp;gt;073104c-4e-4ce-bda-694344ee62&amp;lt;/UID&amp;gt;&amp;lt;consumerSystemId&amp;gt;JTR&amp;lt;/consumerSystemId&amp;gt;&amp;lt;consumerLogin&amp;gt;jbserviceid&amp;lt;/consumerLogin&amp;gt;&amp;lt;logLocation&amp;gt;Successful Completion of Service&amp;lt;/logLocation&amp;gt;&amp;lt;/fedDKPLoggingContext&amp;gt;&lt;/headers&gt;&lt;payload&gt;0&lt;/payload&gt;&lt;/log-message-body&gt;</body></log-event> </code></pre> <p>here is the bash script I wrote.</p> <pre><code>uids=cat $i|grep "Service" |awk 'BEGIN {FS="lt;";RS ="gt;"} {print $2;}'| sort -u for uid in ${uids}; do count=`grep "$uid" test.log|wc -l` if [ "${count}" -ne "0" ]; then unique_uids[counter]="$uid" let counter=counter+1 fi done echo ${unique_uids[@]} echo $counter echo " Unique No:" ${#unique_uids[@]} echo uid StartTime EndTime" > $log for unique_uids in ${unique_uids[@]} ; do responseTime=`cat $i|grep "${unique_uids}" |awk '{split($2,Arr,":|,"); print Arr[1]*3600000+Arr[2]*60000+Arr[3]*1000+Arr[4]}'|sort -n` echo $unique_uids $responseTime >> $log done </code></pre> <p>And the output should be like this Operation comes from id, Consumer comes from documentmetadata and hour is the time 08:54:XX So if we have more than one request and response then need to average of the response times for requests came at that hour.</p> <p>Operation Consumer HOUR Avg-response-time(ms)<br> DKP_DumpDocumentProperties MRP 08 538</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload