Databases
StackOverflow2013
Postsdbo
POIs there any efficient method to find the first byte of all instances of a particular 4 byte block in a file?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIs there any efficient method to find the first byte of all instances of a particular 4 byte block in a file?
    primarykey
    data
    text
    <p>I have files that contains archived binary messages. A small file is around 600MB and contains nearly 9000 messages. Each message begins with a particular four byte flag that I know, which indicates the first four bytes of the message header (and as such must be captured). The message header is a fixed size for all messages. The message header is followed by a payload of a size that is identified in the header. Once I've found the start of a particular message header, I know how many bytes to the end of the header and can use that to extract the number of bytes in the message I need to parse this archive file and isolate each message for processing, making sure that I include all bytes from the first byte of the four byte flag to the end of the specified message length. There is some padding between the messages that varies.</p> <p>Due to the size of the file, I don't want to (and probably can't in all cases) consume the file as a single array. Therefore, I'm looking at things like <code>RandomAccessFile</code> and <code>FileInputStream</code>. It doesn't seem like it's a simple task to scan a file for a particular sequence of bytes and then take every byte from the first byte in that sequence through a known length. <code>RandomAccessFile</code>, especially the <code>read(byte[])</code> and <code>seek()</code> methods seem like they will allow me to implement a solution.</p> <p>To give an idea, my current implementation involves a method called <code>findFlag()</code> that takes a start position in the <code>RandomAccessFile</code>. It seeks to that position and reads the four bytes starting there. If it finds the flag, it returns <code>startPos</code>. Otherwise, it calls itself recursively, moving to <code>startPos + 1</code> and repeats until it finds the flag. Since I know the last byte I read as part of the data message, I would start seeking there:</p> <pre><code>file.seek(startPos); byte[] possibleFlag = new byte[4]; file.read(possibleFlag, 0, possibleFlag.length); if (Arrays.equals(ByteUtils.intToBytes(Message.FLAG), possibleFlag)) { return startPos; } else { return findFlag(startPos + 1); } </code></pre> <p>Am I overlooking something, either in Java (Java 6 or earlier) or in a well-tested external library (such as an Apache library or similar)? If not, are there better solutions for dealing with binary data in Java or any approaches that are particularly well-suited for my problem?</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USThomas Owens
    1. USThomas Owens
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POIs there any efficient method to find the first byte of all instances of a particular 4 byte block in a file?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POIs there any efficient method to find the first byte of all instances of a particular 4 byte block in a file?
      1. USnhahtdh
      1. VTFavorite
    3. VO
      singulars
      1. POIs there any efficient method to find the first byte of all instances of a particular 4 byte block in a file?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COHave you looked at http://stackoverflow.com/questions/644737/are-there-any-java-frameworks-for-binary-file-parsing ? Preon seems to be something you might consider.
      singulars
      1. POIs there any efficient method to find the first byte of all instances of a particular 4 byte block in a file?
      1. USEwald
    2. CO@Ewald Not sure how helpful Preon would be. Until I isolate the messages, there's no consistent format for the file I'm reading. The only given is the fact that the same four byte sequence denotes the start of every message in the file.
      singulars
      1. POIs there any efficient method to find the first byte of all instances of a particular 4 byte block in a file?
      1. USThomas Owens
    3. COI think *something* like what you're currently doing is pretty efficient. Just read in the stream of bytes checking for the marker bytes etc. What else would you want?
      singulars
      1. POIs there any efficient method to find the first byte of all instances of a particular 4 byte block in a file?
      1. USTorious
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload