Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>You're hitting the limit of using the basic shell tools. Assuming about 40 characters per line, File A has 400,000 lines in it and File B has about 1,200,000 lines in it. You're basically running grep for each line in File A and having grep plow through 1,200,000 lines with each execution. that's 480 <strong><em>BILLION</em></strong> lines you're parsing through. Unix tools are surprisingly quick, but even something fast done 480 billion times will add up.</p> <p>You would be better off using a full programming scripting language like Perl or Python. You put all lines in <em>File B</em> in a hash. You take each line in File A, check to see if that fourth field matches something in the hash.</p> <p>Reading in a few hundred thousand lines? Creating a 10,000,000 entry hash? Perl can parse both of those in a matter of minutes.</p> <p>Something -- off the top of my head. You didn't give us much in the way of spects, so I didn't do any testing:</p> <pre><code>#! /usr/bin/env perl use strict; use warnings; use autodie; use feature qw(say); # Create your index open my $file_b, "&lt;", "file_b.txt"; my %index; while (my $line = &lt;$file_b&gt;) { chomp $line; $index{$line} = $line; #Or however you do it... } close $file_b; # # Now check against file_a.txt # open my $file_a, "&lt;", "file_a.txt"; while (my $line = &lt;$file_a&gt;) { chomp $line; my @fields = split /\s+/, $line; if (exists $index{$field[3]}) { say "Line: $line"; } } close $file_a; </code></pre> <p>The hash means you only have to read through file_b once instead of 400,000 times. Start the program, go grab a cup of coffee from the office kitchen. (Yum! non-dairy creamer!) By the time you get back to your desk, it'll be done.</p>
    singulars
    1. This table or related slice is empty.
    1. POBash: How to keep lines in a file that have fields that match lines in another file?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. USDavid W.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI haven't really done much with perl or python before so... yeah. I'll give your script a try, nevertheless. Thanks for the help! I edited the OP so I include what both files look like.
      singulars
      1. PO
      1. USJoe
    2. COWait. I just noticed this thing doesn't actually output anything. What would I have to add on to your script to have it output stuff from file A that passes through? Sorry, don't know perl at all.
      singulars
      1. PO
      1. USJoe
    3. COThis doesn't satisfy the OP's requirement that the match "does NOT have to be an exact match, so if file B had Blah and file A said Blah_blah, it'd be ok".
      singulars
      1. PO
      1. USruakh
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload