Databases
StackOverflow2013
Postsdbo
POPerl - Open large txt file on server and create / save into smaller files of 100 lines each
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPerl - Open large txt file on server and create / save into smaller files of 100 lines each
    primarykey
    data
    text
    <p>I am trying to do this:</p> <ol> <li><p>I FTP a large file of single words (~144,000 and one word per line)</p></li> <li><p>I need to open uploaded file and create files with 100 lines max one word per line (01.txt, 02.txt etc).</p></li> <li><p>I would like the processed 100 to be REMOVED from the original file AFTER the file of 100 is created.</p></li> </ol> <p>The server is shared but, I can install modules if needed. </p> <p>Now, my code below is very crude as my knowledge is VERY limited. One problem is opening the whole file into an array? The shared server does not sport enough memory I assume to open such a large file and read into memory all at once? I just want the first 100 lines. Below is just opening a file that is small enough to be loaded and getting 100 lines into an array. Nothing else. I typed it quickly so, prob has several issues but, show my limited knowledge and need for help.</p> <pre><code> use vars qw($Word @Words $IN); my $PathToFile = '/home/username/public/wordlists/Big-File-Of-Words.txt'; my $cnt= '0'; open $IN, '&lt;', "$PathToFile" or die $!; while (&lt;$IN&gt;) { chomp; $Word = $_; $Word=~ s/\s//g; $Word = lc($Word); ###### if ($cnt &lt;= 99){ push(@Words,$Word); } $cnt++; } close $IN; </code></pre> <p>Thanks so much.</p> <p>Okay, I am trying to implement the code below:</p> <pre><code> #!/usr/bin/perl -w BEGIN { my $b__dir = (-d '/home/username/perl'?'/home/username/perl':( getpwuid($&gt;) )[7].'/perl'); unshift @INC,$b__dir.'5/lib/perl5',$b__dir.'5/lib/perl5/x86_64-linux',map { $b__dir . $_ } @INC; } use strict; use warnings; use CGI; use CGI::Carp qw(fatalsToBrowser warningsToBrowser); print CGI::header(); my $WORD_LIST='/home/username/public/wordlists/Big-File-Of-Words.txt'; sed 's/ *//g' $WORD_LIST | tr '[A-Z]' '[a-z]' | split -l 100 -a6 - words. print 'Done'; 1; </code></pre> <p>But I get:</p> <pre><code>syntax error at split-up-big-file.pl line 12, near "sed 's/ *//g'" Can't find string terminator "'" anywhere before EOF at split-up-big-file.pl line 12. </code></pre> <p><strong>FINALLY:</strong> Well I figured out a quick solution that works. Not pretty:</p> <pre><code> #!/usr/bin/perl -w BEGIN { my $b__dir = (-d '/home/username/perl'?'/home/username/perl':( getpwuid($&gt;) )[7].'/perl'); unshift @INC,$b__dir.'5/lib/perl5',$b__dir.'5/lib/perl5/x86_64-linux',map { $b__dir . $_ } @INC; } use strict; use warnings; use CGI; use CGI::Carp qw(fatalsToBrowser warningsToBrowser); use diagnostics; print CGI::header(); my $sourcefile = '/home/username/public_html/test/bigfile.txt'; my $rowlimit = 100; my $cnt= '1'; open(IN, $sourcefile) or die "Failed to open $sourcefile"; my $outrecno = 1; while(&lt;IN&gt;) { if($outrecno == 1) { my $filename= $cnt.'.txt'; open OUT, "&gt;$filename" or die "Failed to create $filename"; $cnt++; } print OUT $_; if($outrecno++ == $rowlimit) { $outrecno = 1; close FH; } } close FH; </code></pre> <p>I found enough info here to get me going. Thanks...</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USOldDogLearningNewPerlTricks
    1. USOldDogLearningNewPerlTricks
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPerl - Open large txt file on server and create / save into smaller files of 100 lines each
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIf you have shell access, you can do this with the command line tool `split` like this: `split -l 100 /home/username/public/wordlists/Big-File-Of-Words.txt` See the man page here: http://unixhelp.ed.ac.uk/CGI/man-cgi?split
      singulars
      1. POPerl - Open large txt file on server and create / save into smaller files of 100 lines each
      1. USsimbabque
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload