What are the types of streams?

Node.js streams are a powerful mechanism for handling data flow, especially when dealing with large amounts of data or data that arrives in chunks. They allow you to process data piece by piece, without needing to load the entire dataset into memory.

Understanding Node.js Streams

At its core, a stream is an abstract interface for working with streaming data in Node.js. Many built-in Node.js modules implement the stream interface, including fs for file operations, http for network requests, and zlib for compression. Streams are instances of EventEmitter.

The Four Core Stream Types

There are four fundamental types of streams in Node.js, each serving a distinct purpose in data manipulation:

1. Readable Streams

Readable streams are abstractions from which data can be consumed. Examples include fs.createReadStream() for reading files, http.IncomingMessage for incoming HTTP requests on a server, and process.stdin.

const fs = require('fs');
const readableStream = fs.createReadStream('input.txt');

readableStream.on('data', (chunk) => {
  console.log(`Received ${chunk.length} bytes of data.`);
});

readableStream.on('end', () => {
  console.log('Finished reading data.');
});

readableStream.on('error', (err) => {
  console.error('Error:', err);
});

2. Writable Streams

Writable streams are abstractions to which data can be written. Examples include fs.createWriteStream() for writing files, http.ServerResponse for responses from an HTTP server, and process.stdout and process.stderr.

const fs = require('fs');
const writableStream = fs.createWriteStream('output.txt');

writableStream.write('Hello, world!\n');
writableStream.write('This is a writable stream example.\n');
writableStream.end(); // Signifies that no more data will be written

writableStream.on('finish', () => {
  console.log('All data has been written to file.');
});

writableStream.on('error', (err) => {
  console.error('Error:', err);
});

3. Duplex Streams

Duplex streams are both Readable and Writable. They can be read from and written to simultaneously. Examples include TCP sockets (net.Socket), where you can read data sent by the peer and write data to the peer using the same stream object.

const net = require('net');

const server = net.createServer((socket) => {
  // 'socket' is a Duplex stream
  socket.on('data', (data) => {
    console.log('Received from client:', data.toString());
    socket.write('Echo: ' + data.toString()); // Write back to client
  });

  socket.on('end', () => {
    console.log('Client disconnected.');
  });
});

server.listen(3000, () => {
  console.log('Server listening on port 3000');
});

4. Transform Streams

Transform streams are a type of Duplex stream that can modify or transform the data as it is written and then read. They act as a filter, taking input, performing operations, and then providing output. Examples include zlib streams (gzip, deflate) for compression/decompression and crypto streams for encryption/decryption.

const zlib = require('zlib');
const fs = require('fs');

const gzip = zlib.createGzip(); // A Transform stream

const readable = fs.createReadStream('input.txt');
const writable = fs.createWriteStream('input.txt.gz');

readable.pipe(gzip).pipe(writable);

writable.on('finish', () => {
  console.log('File successfully gzipped.');
});

writable.on('error', (err) => {
  console.error('Error during gzip:', err);
});

The `pipe()` Method

The pipe() method is a crucial feature of Node.js streams. It provides a simple way to connect the output of a Readable stream to the input of a Writable stream, automatically handling backpressure to prevent the Writable stream from being overwhelmed by data from the Readable stream. It makes chaining stream operations very straightforward.