たれぱんのびぼーろく

わたしの備忘録、生物学とプログラミングが多いかも

Node.js stream

内部バッファ

internal buffer, internal queue

internal-bufferにデータがたまっていく.
highWaterMark以下の場合、内部的にstream.read()がcallされる.
新しいstreamを作る際、
read()をoverrideし、_read()内でstream.push(chunk)をcallすれば、internal-bufferにchunkがenqueueされる.

generator使ったやり口案.
generatorをstream constructorで作る.
read()内でgenerator.next()をcallする. chunkがyieldされ、yieldされたchunkをstream.push(chunk)する. このread()1サイクルは以上で終了.
次の_read()サイクルでまたgenerator.next()がcallされる. chunk2が(ry. これでstreamの流量制限を利用しながらchunkを渡せる.

When chunk is not null, the chunk of data will be added to the internal queue for users of the stream to consume.

push it into the internal buffer.

If the internal read buffer is below the highWaterMark, and the stream is not currently reading, then calling stream.read(0) will trigger a low-level stream._read() call.

Buffering# Both Writable and Readable streams will store data in an internal buffer that can be retrieved using writable.writableState.getBuffer() or readable.readableState.buffer, respectively.

Data is buffered in Readable streams when the implementation calls stream.push(chunk). If the consumer of the Stream does not call stream.read(), the data will sit in the internal queue until it is consumed.

Once the total size of the internal read buffer reaches the threshold specified by highWaterMark, the stream will temporarily stop reading data from the underlying resource until the data currently buffered can be consumed (that is, the stream will stop calling the internal readable._read() method that is used to fill the read buffer).

Data is buffered in Writable streams when the writable.write(chunk) method is called repeatedly. While the total size of the internal write buffer is below the threshold set by highWaterMark, calls to writable.write() will return true. Once the size of the internal buffer reaches or exceeds the highWaterMark, false will be returned.

複数chunkの同時処理

_writev()

The writable.writev() method may be implemented in addition to writable.write() in stream implementations that are capable of processing multiple chunks of data at once. If implemented, the method will be called with all chunks of data currently buffered in the write queue.
If a stream implementation is capable of processing multiple chunks of data at once, the writable._writev() method should be implemented.
https://nodejs.org/api/stream.html#stream_writable_writev_chunks_callback