Databases
StackOverflow2013
Postsdbo
POC# High speed MD5/SHA hash over network
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POC# High speed MD5/SHA hash over network
    primarykey
    data
    text
    <p>In a C# project that I am currently working on, we're attempting to calculate the MD5 of a large quantity of files over a network (current pot is 2.7 million, client pot may be in excess of 10 million). With the number of files that we are processing, speed is of the issue.</p> <p>The reason we do this is to verify the file was copied to a different location without modification. </p> <p>We currently use the following code to calculate the MD5 of a file</p> <pre><code>MD5 md5 = new MD5CryptoServiceProvider(); StringBuilder sb = new StringBuilder(); byte[] hashMD5 = null; try { // Open stream to file to get MD5 hash for, create hash using (FileStream fsMD5 = new FileStream(sFilePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) hashMD5 = md5.ComputeHash(fsMD5); } catch (Exception ex) { clsLogging.logError(clsLogging.ErrorLevel.ERROR, ex); } string md5sum = ""; if (hashMD5 != null) { // Change hash into readable text foreach (byte hex in hashMD5) sb.Append(hex.ToString("x2")); md5sum = sb.ToString(); } </code></pre> <p>However, the speed of this isn't what my manager has been hoping for. We've gone through a number of changes to the way and number of files that we calculate the MD5 for (i.e. we don't do it for files that we don't copy... until today when my manager changed his mind so <strong>ALL</strong> files must have a MD5 calculated for them, in case at some future time a client wishes to bugger with our program so all files are copied i guess)</p> <p>I realize that the speed of the network is probably a major contributing factor (100Mbit/s). Is there an efficient way to calculate the MD5 of the contents of a file over a network?</p> <p>Thanks in advance. Trevor Watson</p> <p>Edit: put all code in block instead of just a part of it.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USTrevor Watson
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POC# High speed MD5/SHA hash over network
      1. USDaniel
      1. VTFavorite
    2. VO
      singulars
      1. POC# High speed MD5/SHA hash over network
      1. This table or related slice is empty.
      1. VTUpMod
    1. COIs there a reason you can't verify the hashes locally on the machine to which the files were copied?
      singulars
      1. POC# High speed MD5/SHA hash over network
      1. USarootbeer
    2. COFirst point of business is to figure out the bottleneck. Have you ran any benchmarks to see how fast the process takes just reading the file and iterating as you are without computing the hash? Then with the hash; how much overhead is the computation taking? Have you performed the same task locally?
      singulars
      1. POC# High speed MD5/SHA hash over network
      1. USAaron McIver
    3. COI agree with Aaron: The time needed to calculate the hash is the sum of: opening the remote file over network, seeking to the beginning on the drive, reading from drive, streaming contents over network, calculating the hash. Any of it can cause a bottleneck. Some of these can be done in parallel for different files as Andrew Cooper suggested.
      singulars
      1. POC# High speed MD5/SHA hash over network
      1. USwigy
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload