Databases
StackOverflow2013
Postsdbo
POImplementing a buffered transform stream
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POImplementing a buffered transform stream
    primarykey
    data
    text
    <p>I am trying to implement a stream with the <a href="http://nodejs.org/api/stream.html#stream_class_stream_transform_1" rel="noreferrer">new Node.js streams API</a> that will buffer a certain amount of data. When this stream is piped to another stream, or if something consumes <code>readable</code> events, this stream should flush its buffer and then simply become pass-through. The catch is, this stream will be piped to many other streams, and when each destination stream is attached, the buffer must be flushed <em>even if it is already flushed to another stream</em>.</p> <p>For example:</p> <ol> <li><code>BufferStream</code> implements <code>stream.Transform</code>, and keeps a 512KB internal ring buffer</li> <li><code>ReadableStreamA</code> is piped to an instance of <code>BufferStream</code></li> <li><code>BufferStream</code> writes to its ring buffer, reading data from <code>ReadableStreamA</code> as it comes in. (It doesn't matter if data is lost, as the buffer overwrites old data.)</li> <li><code>BufferStream</code> is piped to <code>WritableStreamB</code></li> <li><code>WritableStreamB</code> receives the entire 512KB buffer, and continues to get data as it is written from <code>ReadableStreamA</code> through <code>BufferStream</code>.</li> <li><code>BufferStream</code> is piped to <code>WritableStreamC</code></li> <li><code>WritableStreamC</code> also receives the entire 512KB buffer, but this buffer is now different than what <code>WritableStreamB</code> received, because more data has since been written to <code>BufferStream</code>.</li> </ol> <p>Is this possible with the streams API? The only method I can think of would be to create an object with a method that spins up a new PassThrough stream for each destination, meaning I couldn't simply pipe to and from it.</p> <p>For what it's worth, I've done this with the old "flowing" API by simply listening for new handlers on <code>data</code> events. When a new function was attached with <code>.on('data')</code>, I would call it directly with a copy of the ring buffer.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USBrad
    1. USBrad
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POImplementing a buffered transform stream
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POImplementing a buffered transform stream
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POImplementing a buffered transform stream
      1. This table or related slice is empty.
      1. VTUpMod
    1. COA doubt: does data move only in 512KB bursts or only the first one is 512KB ?
      singulars
      1. POImplementing a buffered transform stream
      1. USuser568109
    2. CO@user568109 When something begins receiving data from the buffered stream, it should receive the initial 512KB buffer (only once) and then will continue to receive data as it passes through the buffered stream as data becomes available. Only the first chunk is 512KB (or whatever the size of the buffer is).
      singulars
      1. POImplementing a buffered transform stream
      1. USBrad
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload