Databases
StackOverflow2013
Postsdbo
POComparing Value of one list inside Gigantic Two Dimen list in python, Fastest way?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POComparing Value of one list inside Gigantic Two Dimen list in python, Fastest way?
    primarykey
    data
    text
    <p>I want to compare if value of one list exist in value of other list.They are huge (50k + items, from database). </p> <h1>EDIT:</h1> <p>I also want to mark the record which is duplicated as duplicate=True and keep them in the table for later refrence.</p> <p>here how the lists are:</p> <pre><code>n_emails=[db_id,checksum for id,checksum in search_results] #I want to compare checksum if exist inside same list or other list and retrieve id (db_id , if exist) #example : n_emails= [[1,'CAFEBABE010'],[2,'bfeafe3df1ds],[3,'deadbeef101'],[5,'CAFEBABE010']] #in this case i want to retrive id 1 and 5 coz they are same checksum for m in n_emails: dups=_getdups(n_emails,m[1],m[0]) n_dups=[casesdb.duplicates.insert( **dup ) for dup in dups] if n_dups: print "Dupe Found" casesdb(casesdb.email_data.id == m[0]).update(duplicated=True) def _getdups(old_lst,em_md5,em_id): dups=[] for old in old_lst: if em_md5==old[0] and old[1]!=em_id: dups.append(dict(org_id=old[1],md5hash=old[0],dupID=em_id,)) return dups </code></pre> <p>But it seems too long and in larger list (50k vs 50k records+) It ran for over 5000 seconds and never done , seems never ending loop? The server i running have 4 GB of ram and 4 cores. Obviously i am doing something wrong.</p> <p>Please help .. thanks a lot!</p> <h1>SOLVED:</h1> <p>Dict Index Mapping is way a lot faster! (When mysql table is not indexed, plese note i have not test against indexed table).</p> <p>Its 20 secs vs 30 miliseconds = 20*1000 / 30 = 666 Times! LOL</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USPhyo Arkar Lwin
    1. USPhyo Arkar Lwin
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POComparing Value of one list inside Gigantic Two Dimen list in python, Fastest way?
      1. USunutbu
      1. VTFavorite
    2. VO
      singulars
      1. POComparing Value of one list inside Gigantic Two Dimen list in python, Fastest way?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POComparing Value of one list inside Gigantic Two Dimen list in python, Fastest way?
      1. USPhyo Arkar Lwin
      1. VTFavorite
    1. COIs there are reason why you would want to do this in code instead of via a database query?
      singulars
      1. POComparing Value of one list inside Gigantic Two Dimen list in python, Fastest way?
      1. USRyan Ginstrom
    2. CODo your want to remove the duplicated values? Because there should be more effective ways if you put your final demands.
      singulars
      1. POComparing Value of one list inside Gigantic Two Dimen list in python, Fastest way?
      1. USKabie
    3. COWhat i want to do is to get Duplicated Entries, and build a table holding them, which will be refrenced later. Lets say When user browse a record , he/she will be shown with a list of duplicated entries so he/she can also check them (or skip them). Mysql's distinct will eliminate duplicates right?
      singulars
      1. POComparing Value of one list inside Gigantic Two Dimen list in python, Fastest way?
      1. USPhyo Arkar Lwin
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload