たれぱんのびぼーろく

わたしの備忘録、生物学とプログラミングが多いかも

Node.js stream

ReadableとWritable

Sourceからreadして処理をしていくので、Readable streamがSource相当.
writeしてあとはおまかせーと吸い込んでいくので、Writable streamがSink, distination相当.

Readable streams are an abstraction for a source from which data is consumed.
Writable streams are an abstraction for a destination to which data is written.

Readableの使い方

  1. callbackをセットして、callbackを自動でcallしてもらう
  2. Writable streamとつなぐ
  3. .read()を手動でcallして、出てきたデータをつかう

Readableの詳細

2つのモード: flowing, paused
flowingモードだと、イベントを発火しながらデータを吐き出し続ける
pausedモードだと、.read()をcallしてデータを取り出す
デフォルトではpausedモード.

Readable streams effectively operate in one of two modes: flowing and paused.
provided to an application as quickly as possible using events
In paused mode, the stream.read() method must be called explicitly to read chunks of data from the stream.
All Readable streams begin in paused mode

.read()はどれくらい取り出すか指定できる

If the size argument is not specified, all of the data contained in the internal buffer will be returned.

.read()は推奨されてない

In general, it is recommended that developers avoid the use of the 'readable' event and the readable.read() method in favor of using either readable.pipe() or the 'data' event.

Writableとcallback()

writable.write()

writable.write()でデータの書き込み。
internal bufferがいっぱいになるとfalseを返してくる。無理やり書き込んでもinternal bufferにバッファーされるが、当然良くないよ。いっぱいになったのち、internal bufferが消費されて空っぽになると "drain"イベントが発火される.

The return value is true if the internal buffer is less than the highWaterMark configured when the stream was created after admitting chunk. If false is returned, further attempts to write data to the stream should stop until the 'drain' event is emitted.
無理やり書き込んでも…
While a stream is not draining, calls to write() will buffer chunk, and return false.
再開は
Once all currently buffered chunks are drained (accepted for delivery by the operating system), the 'drain' event will be emitted.

writable._write()

._write(chunk, encoding, callback)の形でcallbackを取る.
callbackは"write"の成功または失敗を伝えるために呼び出される.

The callback method must be called to signal either that the write completed successfully or failed with an error.

._write()の実行 -> callbackの呼び出しとなるが、callbackが呼び出される前に追加で.write()がcallされた場合、.write()のデータはinternal bufferにバッファーされる.
callback()がcallされると"drain"イベントが発火される.

It is important to note that all calls to writable.write() that occur between the time writable._write() is called and the callback is called will cause the written data to be buffered.
Once the callback is invoked, the stream will emit a 'drain' event.

callbackの使い方

成功時には第一引数にnullを、失敗時には第一引数にError objectを渡す。

The first argument passed to the callback must be the Error object if the call failed or null if the write succeeded.

内部バッファ

internal buffer, internal queue

internal-bufferにデータがたまっていく.
highWaterMark以下の場合、内部的にstream.read()がcallされる.
新しいstreamを作る際、
read()をoverrideし、_read()内でstream.push(chunk)をcallすれば、internal-bufferにchunkがenqueueされる.

generator使ったやり口案.
generatorをstream constructorで作る.
read()内でgenerator.next()をcallする. chunkがyieldされ、yieldされたchunkをstream.push(chunk)する. このread()1サイクルは以上で終了.
次の_read()サイクルでまたgenerator.next()がcallされる. chunk2が(ry. これでstreamの流量制限を利用しながらchunkを渡せる.

When chunk is not null, the chunk of data will be added to the internal queue for users of the stream to consume.

push it into the internal buffer.

If the internal read buffer is below the highWaterMark, and the stream is not currently reading, then calling stream.read(0) will trigger a low-level stream._read() call.

Buffering# Both Writable and Readable streams will store data in an internal buffer that can be retrieved using writable.writableState.getBuffer() or readable.readableState.buffer, respectively.

Data is buffered in Readable streams when the implementation calls stream.push(chunk). If the consumer of the Stream does not call stream.read(), the data will sit in the internal queue until it is consumed.

Once the total size of the internal read buffer reaches the threshold specified by highWaterMark, the stream will temporarily stop reading data from the underlying resource until the data currently buffered can be consumed (that is, the stream will stop calling the internal readable._read() method that is used to fill the read buffer).

Data is buffered in Writable streams when the writable.write(chunk) method is called repeatedly. While the total size of the internal write buffer is below the threshold set by highWaterMark, calls to writable.write() will return true. Once the size of the internal buffer reaches or exceeds the highWaterMark, false will be returned.

複数chunkの同時処理

_writev()

The writable.writev() method may be implemented in addition to writable.write() in stream implementations that are capable of processing multiple chunks of data at once. If implemented, the method will be called with all chunks of data currently buffered in the write queue.
If a stream implementation is capable of processing multiple chunks of data at once, the writable._writev() method should be implemented.
https://nodejs.org/api/stream.html#stream_writable_writev_chunks_callback