Databases
StackOverflow2013
Postsdbo
POReading stream line by line without knowing its encoding
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POReading stream line by line without knowing its encoding
    primarykey
    data
    text
    <p>I have a situation where I need to process some data from a stream line by line. The problem is that the encoding of the data is not known in advance; it might be <code>UTF-8</code> or any legacy single-byte encoding (e.g. <code>Latin1</code>, <code>ISO-8859-5</code>, etc). It will <em>not</em> be <code>UTF16</code> or exotics like <code>EBCDIC</code>, so I can reasonably expect <code>\n</code> to be unambiguous, so in theory I can split it into lines. At some point, when I encounter an empty line, I will need to feed the rest of the stream somewhere else (without splitting it into lines, but still without any reencoding); think in terms of HTTP-style headers followed by an opaque body.</p> <p>Here is what I got:</p> <pre><code>function processStream(stream) { var buffer = ''; function splitLines(data) { buffer += data; var lf = buffer.indexOf('\n'); while (lf &gt;= 0) { var line = buffer.substr(0, lf - 1); buffer = buffer.substr(lf + 1); this.emit('line', line); lf = buffer.indexOf('\n'); } } function processHeader(line) { if (line.length) { // do something with the line } else { // end of headers, stop splitting lines and start processing the body this .removeListener('data', splitLines) .removeAllListeners('line') .on('data', processBody); if (buffer.length) { // process leftover buffer as part of the body processBody(buffer); buffer = ''; } } } function processBody(data) { // do something with the body chunks } stream.setEncoding('binary'); stream .on('data', splitLines) .on('line', processHeader); } </code></pre> <p>It does the job, but the problem is that the <code>binary</code> encoding is deprecated and will probably disappear in the future, leaving me without that option. All other <code>Buffer</code> encodings will either mangle the data or fail to decode it altogether if (most likely, when) it does not match the encoding. Working with <code>Uint8Array</code> instead will mean slow and inconvenient Javascript loops over the data just to find a newline.</p> <p>Any suggestions on how to split a stream into lines on the fly, while remaining encoding-agnostic <em>without</em> using <code>binary</code> encoding?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USlanzz
    1. USlanzz
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POReading stream line by line without knowing its encoding
      1. USlanzz
      1. VTBountyStart
    2. VO
      singulars
      1. POReading stream line by line without knowing its encoding
      1. This table or related slice is empty.
      1. VTUpMod
    1. COWhy don't you use 'hex' data instead of binary? You say all other encodings mangle the data, but anything represented in binary should be able to be represented in hex. Granted you might get up to 7 bits of padding if your data is not represented in octets.
      singulars
      1. POReading stream line by line without knowing its encoding
      1. USDeadAlready
    2. COBecause that will effectively use double the amount of memory for each data chunk, it will require unnecessary conversions to and from hex just for the sake of splitting lines, and also searching for a `0A` substring will need an additional check if it falls on an even offset into the string and might need a second search if it doesn't. Overall, it seems neither clean nor effective. We're talking about actual textual data here, it is not binary; it is just that I'm handling it in `binary` encoding for the sake of not doing any I/O conversion on it.
      singulars
      1. POReading stream line by line without knowing its encoding
      1. USlanzz
    3. CO(and yes, I know that `binary` actually involves conversion, but it preserves the data in a representation that can be reversed into the original bytes)
      singulars
      1. POReading stream line by line without knowing its encoding
      1. USlanzz
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload