Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <h2>Faster algorithm</h2> <p>In case many files of 'big size' should be analyzed (images, mp3, pdf documents), it would be interesting/faster to have the following comparison algorithm:</p> <ol> <li><p>a first fast hash is performed on the first N bytes of the file (say 1KB). This hash would say if files are different without doubt, but will not say if two files are exactly the same (accuracy of the hash, limited data read from disk)</p></li> <li><p>a second, slower, hash, which is more accurate and performed on the whole content of the file, if a collision occurs in the first stage</p></li> </ol> <p>Here is an implementation of this algorithm:</p> <pre><code>import hashlib def Checksum(current_file_name, check_type = 'sha512', first_block = False): """Computes the hash for the given file. If first_block is True, only the first block of size size_block is hashed.""" size_block = 1024 * 1024 # The first N bytes (1KB) d = {'sha1' : hashlib.sha1, 'md5': hashlib.md5, 'sha512': hashlib.sha512} if(not d.has_key(check_type)): raise Exception("Unknown checksum method") file_size = os.stat(current_file_name)[stat.ST_SIZE] with file(current_file_name, 'rb') as f: key = d[check_type].__call__() while True: s = f.read(size_block) key.update(s) file_size -= size_block if(len(s) &lt; size_block or first_block): break return key.hexdigest().upper() def find_duplicates(files): """Find duplicates among a set of files. The implementation uses two types of hashes: - A small and fast one one the first block of the file (first 1KB), - and in case of collision a complete hash on the file. The complete hash is not computed twice. It flushes the files that seems to have the same content (according to the hash method) at the end. """ print 'Analyzing', len(files), 'files' # this dictionary will receive small hashes d = {} # this dictionary will receive full hashes. It is filled # only in case of collision on the small hash (contains at least two # elements) duplicates = {} for f in files: # small hash to be fast check = Checksum(f, first_block = True, check_type = 'sha1') if(not d.has_key(check)): # d[check] is a list of files that have the same small hash d[check] = [(f, None)] else: l = d[check] l.append((f, None)) for index, (ff, checkfull) in enumerate(l): if(checkfull is None): # computes the full hash in case of collision checkfull = Checksum(ff, first_block = False) l[index] = (ff, checkfull) # for each new full hash computed, check if their is # a collision in the duplicate dictionary. if(not duplicates.has_key(checkfull)): duplicates[checkfull] = [ff] else: duplicates[checkfull].append(ff) # prints the detected duplicates if(len(duplicates) != 0): print print "The following files have the same sha512 hash" for h, lf in duplicates.items(): if(len(lf)==1): continue print 'Hash value', h for f in lf: print '\t', f.encode('unicode_escape') if \ type(f) is types.UnicodeType else f return duplicates </code></pre> <p>The <code>find_duplicates</code> function takes a list of files. This way, it is also possible to compare two directories (for instance, to better synchronize their content.) An example of function creating a list of files, with specified extension, and avoiding entering in some directories, is below:</p> <pre><code>def getFiles(_path, extensions = ['.png'], subdirs = False, avoid_directories = None): """Returns the list of files in the path :'_path', of extension in 'extensions'. 'subdir' indicates if the search should also be performed in the subdirectories. If extensions = [] or None, all files are returned. avoid_directories: if set, do not parse subdirectories that match any element of avoid_directories.""" l = [] extensions = [p.lower() for p in extensions] if not extensions is None \ else None for root, dirs, files in os.walk(_path, topdown=True): for name in files: if(extensions is None or len(extensions) == 0 or \ os.path.splitext(name)[1].lower() in extensions): l.append(os.path.join(root, name)) if(not subdirs): while(len(dirs) &gt; 0): dirs.pop() elif(not avoid_directories is None): for d in avoid_directories: if(d in dirs): dirs.remove(d) return l </code></pre> <p>This method is convenient for not parsing <code>.svn</code> paths for instance, which surely will trigger colliding files in <code>find_duplicates</code>. </p> <p>Feedbacks are welcome.</p>
    singulars
    1. This table or related slice is empty.
    1. POFinding duplicate files and removing them
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USRaffi
    1. USRaffi
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload