StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow to improve perl script performance?
text
Body
copied!<p>I am running the ucm2.pl script to scan a huge directory structure (directory is a network drive mapped to local). I have two perl scripts ucm1.pl and ucm2.pl. I am running ucm2.pl parellely for different arguments and it is called through ucm1.pl.</p> <p>ucm1.pl -</p> <pre><code> #!/usr/bin/perl use strict; use warnings; use Parallel::ForkManager; my $filename ="intfSplitList.txt"; #(this will have list of all the input files. eg intfSplit_0....intfSplit_50) my $lines; my $buffer; open(FILE, $filename) or die "Can't open `$filename': $!"; while (<FILE>) { $lines = $.; } close FILE; print "The number of lines in $filename is $lines \n"; my $pm = Parallel::ForkManager->new($lines); #(it will set the no. of parallel processes) open (my $fh, '<', "intfSplitList.txt") or die $!; while (my $data = <$fh>) { chomp $data; my $pid = $pm->start and next; system ("perl ucm2.pl -iinput.txt -f$data"); #(call the ucm2.pl) #(input.txt file will have search keyword and $data will have intfSplit_*.txt files) $pm->finish; # Terminates the child process } </code></pre> <p>ucm2.pl code-</p> <pre><code>#!/usr/bin/perl use strict; use warnings; use File::Find; use Getopt::Std; #getting the input parameters getopts('i:f:'); our($opt_i, $opt_f); my $searchKeyword = $opt_i; #Search keyword file. my $intfSplit = $opt_f; #split file my $path = "Z:/aims/"; #source directory my $searchString; #search keyword open FH, ">log.txt"; #open the log file to write print FH "$intfSplit ". "started at ".(localtime)."\n"; #write the log file open (FILE,$intfSplit); #open the split file to read while(<FILE>){ my $intf= $_; #setting the interface to intf chomp($intf); my $dir = $path.$intf; chomp($dir); print "$dir \n"; open(INP,$searchKeyword); #open the search keyword file to read while (<INP>){ $searchString =$_; #setting the search keyword to searchString chomp($searchString); print "$searchString \n"; open my $out, ">", "vob$intfSplit.txt" or die $!; #open the vobintfSplit_* file to write #calling subroutine printFile to find and print the path of element find(\&printFile,$dir); #the subroutine will search for the keyword and print the path if keyword is exist in file. sub printFile { my $element = $_; if(-f $element && $element =~ /\.*$/){ open my $in, "<", $element or die $!; while(<$in>) { if (/\Q$searchString\E/) { my $last_update_time = (stat($element))[9]; my $timestamp = localtime($last_update_time); print $out "$File::Find::name". " $timestamp". " $searchString\n"; last; } } } } } } print FH "$intfSplit ". "ended at ".(localtime)."\n"; #write the log file </code></pre> <p>everything is running fine but its running for very long time for single keyword search also. can anyone please suggest some better way to improve the performance.</p> <p>Thanks in advance!!</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload