Databases
StackOverflow2013
Postsdbo
POFileChannel ByteBuffer and Hashing Files
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POFileChannel ByteBuffer and Hashing Files
    primarykey
    data
    text
    <p>I built a file hashing method in java that takes input string representation of a <code>filepath+filename</code> and then calculates the hash of that file. The hash can be any of the native supported java hashing algo's such as <code>MD2</code> through <code>SHA-512</code>. </p> <p>I am trying to eek out every last drop of performance since this method is an integral part of a project I'm working on. I was advised to try using <code>FileChannel</code> instead of a regular <code>FileInputStream</code>. </p> <p>My original method:</p> <pre><code> /** * Gets Hash of file. * * @param file String path + filename of file to get hash. * @param hashAlgo Hash algorithm to use. &lt;br/&gt; * Supported algorithms are: &lt;br/&gt; * MD2, MD5 &lt;br/&gt; * SHA-1 &lt;br/&gt; * SHA-256, SHA-384, SHA-512 * @return String value of hash. (Variable length dependent on hash algorithm used) * @throws IOException If file is invalid. * @throws HashTypeException If no supported or valid hash algorithm was found. */ public String getHash(String file, String hashAlgo) throws IOException, HashTypeException { StringBuffer hexString = null; try { MessageDigest md = MessageDigest.getInstance(validateHashType(hashAlgo)); FileInputStream fis = new FileInputStream(file); byte[] dataBytes = new byte[1024]; int nread = 0; while ((nread = fis.read(dataBytes)) != -1) { md.update(dataBytes, 0, nread); } fis.close(); byte[] mdbytes = md.digest(); hexString = new StringBuffer(); for (int i = 0; i &lt; mdbytes.length; i++) { hexString.append(Integer.toHexString((0xFF &amp; mdbytes[i]))); } return hexString.toString(); } catch (NoSuchAlgorithmException | HashTypeException e) { throw new HashTypeException("Unsuppored Hash Algorithm.", e); } } </code></pre> <p>Refactored method: </p> <pre><code> /** * Gets Hash of file. * * @param file String path + filename of file to get hash. * @param hashAlgo Hash algorithm to use. &lt;br/&gt; * Supported algorithms are: &lt;br/&gt; * MD2, MD5 &lt;br/&gt; * SHA-1 &lt;br/&gt; * SHA-256, SHA-384, SHA-512 * @return String value of hash. (Variable length dependent on hash algorithm used) * @throws IOException If file is invalid. * @throws HashTypeException If no supported or valid hash algorithm was found. */ public String getHash(String fileStr, String hashAlgo) throws IOException, HasherException { File file = new File(fileStr); MessageDigest md = null; FileInputStream fis = null; FileChannel fc = null; ByteBuffer bbf = null; StringBuilder hexString = null; try { md = MessageDigest.getInstance(hashAlgo); fis = new FileInputStream(file); fc = fis.getChannel(); bbf = ByteBuffer.allocate(1024); // allocation in bytes int bytes; while ((bytes = fc.read(bbf)) != -1) { md.update(bbf.array(), 0, bytes); } fc.close(); fis.close(); byte[] mdbytes = md.digest(); hexString = new StringBuilder(); for (int i = 0; i &lt; mdbytes.length; i++) { hexString.append(Integer.toHexString((0xFF &amp; mdbytes[i]))); } return hexString.toString(); } catch (NoSuchAlgorithmException e) { throw new HasherException("Unsupported Hash Algorithm.", e); } } </code></pre> <p>Both return a correct hash, however the refactored method only seems to cooperate on small files. When i pass in a large file, it completely chokes out and I can't figure out why. I'm new to <code>NIO</code> so please advise.</p> <p>EDIT: Forgot to mention I'm throwing SHA-512's through it for testing.</p> <p><code>UPDATE:</code> Updating with my now current method.</p> <pre><code> /** * Gets Hash of file. * * @param file String path + filename of file to get hash. * @param hashAlgo Hash algorithm to use. &lt;br/&gt; * Supported algorithms are: &lt;br/&gt; * MD2, MD5 &lt;br/&gt; * SHA-1 &lt;br/&gt; * SHA-256, SHA-384, SHA-512 * @return String value of hash. (Variable length dependent on hash algorithm used) * @throws IOException If file is invalid. * @throws HashTypeException If no supported or valid hash algorithm was found. */ public String getHash(String fileStr, String hashAlgo) throws IOException, HasherException { File file = new File(fileStr); MessageDigest md = null; FileInputStream fis = null; FileChannel fc = null; ByteBuffer bbf = null; StringBuilder hexString = null; try { md = MessageDigest.getInstance(hashAlgo); fis = new FileInputStream(file); fc = fis.getChannel(); bbf = ByteBuffer.allocateDirect(8192); // allocation in bytes - 1024, 2048, 4096, 8192 int b; b = fc.read(bbf); while ((b != -1) &amp;&amp; (b != 0)) { bbf.flip(); byte[] bytes = new byte[b]; bbf.get(bytes); md.update(bytes, 0, b); bbf.clear(); b = fc.read(bbf); } fis.close(); byte[] mdbytes = md.digest(); hexString = new StringBuilder(); for (int i = 0; i &lt; mdbytes.length; i++) { hexString.append(Integer.toHexString((0xFF &amp; mdbytes[i]))); } return hexString.toString(); } catch (NoSuchAlgorithmException e) { throw new HasherException("Unsupported Hash Algorithm.", e); } } </code></pre> <p>So I attempted a benchmark hashing out the MD5 of a 2.92GB file using my original example and my latest update's example. Of course any benchmark is relative since there is OS and disk caching and other "magic" going on that will skew repeated reads of the same files... but here's a shot at some benchmarks. I loaded each method up and fired it off 5 times after compiling it fresh. The benchmark was taken from the last (5th) run as this would be the "hottest" run for that algorithm, and any "magic" (in my theory anyways).</p> <pre><code>Here's the benchmarks so far: Original Method - 14.987909 (s) Latest Method - 11.236802 (s) </code></pre> <p>That is a <code>25.03% decrease</code> in time taken to hash the same 2.92GB file. Pretty good.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USSnakeDoc
    1. USSnakeDoc
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POFileChannel ByteBuffer and Hashing Files
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POFileChannel ByteBuffer and Hashing Files
      1. USvishal_aim
      1. VTFavorite
    3. VO
      singulars
      1. POFileChannel ByteBuffer and Hashing Files
      1. This table or related slice is empty.
      1. VTUpMod
    1. COWhy not use [MessageDigest.update(ByteBuffer)](http://docs.oracle.com/javase/6/docs/api/java/security/MessageDigest.html#update%28java.nio.ByteBuffer%29) method that takes `ByteBuffer` directly instead of using the backing array?
      singulars
      1. POFileChannel ByteBuffer and Hashing Files
      1. USprunge
    2. COJust wanted to add for future visitors -- if you switch to using `ByteBuffer.allocateDirect()` then there is no backing array and `ByteBuffer.array()` will `fail`. Instead switch over to using `MessageDigest.update(ByteBuffer)` per @prunge advice. This is not only more efficient, but cleaner than trying to read the buffer to some array then pass that array into `MessageDigest.update()`.
      singulars
      1. POFileChannel ByteBuffer and Hashing Files
      1. USSnakeDoc
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload