Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p><strong>Use SQL</strong> </p> <ul> <li>Create a table with the same columns as your CSV files.</li> <li>Insert the data from the first CSV file.</li> <li>Add indexes to speed up queries.</li> <li>Compare with other CSV files by reading a line and issuing a SELECT.</li> </ul> <p>You did not describe how you compare n files, and there are several ways to do so. If you just want to find the line that are in A1 but not in A2,...,An, then you'll just have to add a boolean column <em>diff</em> in your table. If you want to know in which files a line is repeated, you'll need a text column, or a new table if a line can be in several files.</p> <p><em>Edit:</em> a few words on performance if you're using MySQL (I do not now much about other RDBMS).</p> <p>Inserting lines one by one would be too slow. You probably can't use <code>LOAD DATA</code> unless you can put the CSV files directly onto the DB server's filesystem. So I guess the best solution is to read a few hundreds of lines in the CSV then send a multiple insert query <code>INSERT INTO mytable VALUES (..1..), (..2..)</code>.</p> <p>You can't issue a <code>SELECT</code> for each line you read in your other files, so you'd better put them in another table. Then issue a multiple-table update to mark the rows that are identical in the tables t1 and t2: <code>UPDATE t1 JOIN t2 ON (t1.a = t2.a AND t1.b = t2.b) SET t1.diff=1</code></p> <p>Maybe you could try using sqlite. No concurrency problems here, and it could be faster than the client/server model of MySQL. And you don't need to setup much to use sqlite.</p>
    singulars
    1. This table or related slice is empty.
    1. POComparing multiple very large csv files against each other
      singulars
      1. PTQuestion
    1. PTAnswer
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POComparing multiple very large csv files against each other
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThanks. I need to compare if A1 is in file 2/3/4 on any position and so on. I guess I will give it a try with a temporary database
      singulars
      1. PO
      1. USTalisin
    2. COYeah I use LOAD DATA to import the csv. Its still slow but its ok for now. Thanks dude.
      singulars
      1. PO
      1. USTalisin
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload