StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>I modified my code to build a Hash as suggested. I have not incorporate the output to binary yet due to time limitations. Plus I need to figure out how to reference the hash to get the data out and pack it into binary. I don't think that part should be to difficult ... hopefully</p> <p>On an actual data file (~350MB & 2.0 Million lines) the following code takes approximately 3 minutes to build the hash. CPU usage was 100% on 1 of my cores (nill on the other 3) and Perl memory usage topped out at around 325MB ... until it dumped millions of lines to the prompt. However the print Dump will be replaced with a binary pack.</p> <p>Please let me know if I am making any rookie mistakes.</p> <pre><code>#!/usr/bin/perl use strict; use warnings; use Data::Dumper; my $lineArg1 = $ARGV[0]; open(INFILE, $lineArg1); my $line; my @param_names; my @template; while ($line = <INFILE>) { chomp $line; #Remove New Line if ($line =~ s/\s+filter = ALL_VALUES//) { #Find parameters and build a list push @param_names, trim($line); } elsif ($line =~ /^----/) { @template = map {'A'.length} $line =~ /(\S+\s*)/g; #Make template for unpack $template[-1] = 'A*'; my $data_start_pos = tell INFILE; last; #Reached start of data exit loop } } my $size = $#param_names+1; my @getType = ((1) x $size); my $template = "@template"; my @lineData; my %dataHash; my $lineCount = 0; while ($line = <INFILE>) { if ($lineCount % 100000 == 0){ print "On Line: ".$lineCount."\n"; } if ($line =~ /^\d/) { chomp($line); @lineData = unpack $template, $line; my ($inHeader, $headerIndex) = findStr($lineData[1], @param_names); if ($inHeader) { push @{$dataHash{$lineData[1]}{time} }, $lineData[0]; push @{$dataHash{$lineData[1]}{data} }, $lineData[3]; if ($getType[$headerIndex]){ # Things that only need written once $dataHash{$lineData[1]}{type} = $lineData[2]; $getType[$headerIndex] = 0; } } } $lineCount ++; } # END WHILE <INFILE> close(INFILE); print Dumper \%dataHash; #WRITE BINARY FILE and TOC FILE my %convert = (TXT=>sub{pack 'A*', join "\n", @_}, D=>sub{pack 'd*', @_}, UI=>sub{pack 'L*', @_}); open my $binfile, '>:raw', $lineArg1.'.bin'; open my $tocfile, '>', $lineArg1.'.toc'; for my $param (@param_names){ my $data = $dataHash{$param}; my @toc_line = ($param, $data->{type}, tell $binfile ); print {$binfile} $convert{D}->(@{$data->{time}}); push @toc_line, tell $binfile; print {$binfile} $convert{$data->{type}}->(@{$data->{data}}); push @toc_line, tell $binfile; print {$tocfile} join(',',@toc_line,''),"\n"; } sub trim { #Trim leading and trailing white space my (@strings) = @_; foreach my $string (@strings) { $string =~ s/^\s+//; $string =~ s/\s+$//; chomp ($string); } return wantarray ? @strings : $strings[0]; } # END SUB sub findStr { #Return TRUE if string is contained in array. my $searchStr = shift; my $i = 0; foreach ( @_ ) { if ($_ eq $searchStr){ return (1,$i); } $i ++; } return (0,-1); } # END SUB </code></pre> <p>The output is as follows:</p> <pre><code>$VAR1 = { 'Param 1' => { 'time' => [ '1.1', '3.2', '5.3' ], 'type' => 'UI', 'data' => [ '5', '10', '15' ] }, 'Param 2' => { 'time' => [ '4.5', '6.121' ], 'type' => 'D', 'data' => [ '2.1234', '3.1234' ] }, 'Param 3' => { 'time' => [ '2.23', '7.56' ], 'type' => 'TXT', 'data' => [ 'Some Text 1', 'Some Text 2' ] } }; </code></pre> <p>Here is the output TOC File:</p> <pre><code>Param 1,UI,0,24,36, Param 2,D,36,52,68, Param 3,TXT,68,84,107, </code></pre> <p>Thanks everyone for their help so far! This is an excellent resource!</p> <p>EDIT: Added Binary & TOC file writing code. </p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload