Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>I would say that when hashing files, your bottleneck is most likely to be file I/O and not the hashing algorithm itself. Obviously, this observation holds unless you select a <em>very</em> slow hash algorithm (I know of none in mainstream use that are that slow) or your hardware is very unusual and peculiar.</p> <p>To give you some (very inaccurate) numbers, most good implementations of cryptographic hash algorithms (which are probably much stronger than you need) run at many hundreds of megabytes or a couple of gigabytes per second, but most common disk systems run at tens of megabytes or a couple of hundred megabytes per second.</p> <p>Note that in these (very rough) numbers, I'm ignoring the overhead of filesystem and OS, and also I'm assuming that you'll use a cryptographic hash algorithm (which are typically slower than simpler ones.)</p> <p>All in all, my advice is that you just pick something like MD5 or SHA-1 or SHA-256 (these are all cryptographic hashes) which have a lot of good, fast and free implementations and practically remove any chance of collisions. And instead of picking a fast but inferior hash algorithm, optimize your I/O (by doing asynchronous/non-blocking reading of your files) because that's where the bottleneck is going to be.</p> <p>Here are just a few crypto libraries for C/C++ that implement various hash functions, and have licenses suitable for your use case:</p> <ol> <li><a href="http://cryptopp.com/">Crypto++</a></li> <li><a href="http://botan.randombit.net/">Botan</a></li> <li><a href="http://www.openssl.org/">OpenSSL</a></li> </ol>
    singulars
    1. This table or related slice is empty.
    1. POWhat hash algorithm to use for quick file hashing?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USyzt
    1. USyzt
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POWhat hash algorithm to use for quick file hashing?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COConcentrating on the overheads of the file IO is a good suggestion. Mem-mapping is a good thing to look into too to solve this.
      singulars
      1. PO
      1. USWill
    2. CO@Will: Certainly memory-mapping the input files can help, but I think for this particular use case (reading files sequentially and from start to finish,) using I/O tricks may yield faster results.
      singulars
      1. PO
      1. USyzt
    3. CO@Will: (in continuation of my last comment!) Oh, unless you map the file into memory, and start hashing from the beginning but systematically touch memory pages ahead to give to OS time to generate the page faults and load the content into memory! That's certainly easier to do than to mess with (e.g.) Windows overlapped I/O API or completion ports or spawning multiple threads to do I/O!
      singulars
      1. PO
      1. USyzt
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload