Or how an image loading in a browser eventually led to five terabytes a day, oil tankers, and a rather suspicious collection of encrypted SD cards.
I can’t remember exactly when I first encountered Node.js. It was early enough that asynchronous programming frequently meant nesting callbacks until the code resembled an archaeological excavation site. I wasn’t using Node professionally. I was just playing with it, and one experiment involved streaming an image to a browser.
The image appeared progressively, from top to bottom, as the bytes arrived. I was used to applications that waited and then presented the finished thing. Here was a browser doing useful work before the operation had finished. I was blown away!
Before anyone rushes to explain that Apache could stream too: yes, we know. What fascinated me wasn’t a server capability. It was the way I’d been thinking about work. Why did I keep treating finished as a prerequisite for started?
I began organising Node applications around events, with named handlers I could actually follow rather than callbacks nesting into the Earth’s crust. I didn’t know anything about functional programming at the time. I just liked what happened when the data was allowed to move.
You’re building a bloody dam again!
This eventually became a phrase I’d throw at fellow engineers: “You’re building a bloody dam again!” Usually when somebody read an entire payload into memory just to do something that could have been done as it arrived. Lol.
Think of a river. If we insist on filling a reservoir before using a single drop, we’ve turned a continuous flow into a batch. Sometimes collecting everything first is necessary. But if the job is to inspect, filter or transform what passes by, why make the whole river wait? That’s the bloody dam I’m complaining about.

Follow the data
Watch eight complete records move through the same pipeline. The difference is when we allow work to begin.
A complete record can be processed without waiting for the rest of the payload.
Conceptual illustration. We are comparing an explicit buffering decision with incremental record processing; a Promise does not itself buffer data.
Now picture airport immigration. Travellers arrive, an officer checks each passport, and each traveller moves on. Nobody waits for every flight of the day to land before opening the first passport. That’s the useful checkpoint: validate one complete record, then let it through. If the counter can’t keep up, the queue slows the arrivals. If there’s more independent work than one counter can handle, we can open another.

That’s really the difference. We can wait for everything to arrive, or start working as it comes in. Promises aren’t the villain here. They tell us when a job has finished. Streams let us work while it’s still happening. Quite often, we want both.
Early Node made that distinction wonderfully visible. Its HTTP response API let us write() repeatedly and end() when we were done. Yet a familiar pattern in early web tooling was to collect all the data events into a buffer and hand it to a callback only at end. Here’s the shape of it, from Express 0.10.0’s HTTP client:
res.on('data', function (chunk) {
buf += chunk;
});
res.on('end', function () {
callback(null, buf, res);
}); Sometimes we really do need the complete response. Fair enough. But here the decision has already been made for us. Nobody gets to touch the data until the last chunk arrives. A perfectly good stream has become a waiting room.
Five terabytes. One M2.
It really, really, really shines with transform streams. I once had to turn terabytes of JSON logs into CSV for the BI team at DBS, while working on its marketplace platform. Around 5TB a day. The official route involved a military hierarchy of requests to commission high-end AWS machines. By the time those approvals arrived, I’d have missed the deadline.
So I tried it on an M2. Believe it or not, I was sceptical myself. Then it worked. Not only did it work, the bloody thing was practically flying.
The trick wasn’t some exotic algorithm. It was refusing to treat five terabytes as one object. Read the input incrementally, identify complete records, validate and transform them, and keep moving. The output could branch: CSV for the BI team on one side, compressed archives for S3 on the other. No need to hold the whole dataset hostage while waiting for the last byte.
Of course, splitting a stream into two destinations needs a little care. What if S3 slows down while the CSV writer races ahead? Both branches need their own flow control and error handling. Get that right, though, and we can produce both outputs in one pass without swallowing the entire dataset first.
When I tell other engineers this story, I often get a suspicious look. Five terabytes? On that machine? I understand. I didn’t quite believe it until I watched it happen. We’re so accustomed to equating large datasets with large fleets of hardware that we forget the distinction between the size of the whole dataset and the size of the working set.
Then someone gave me more cores
Later, at Chord X, I had to process terabytes of IoT data from oil tankers. Same shit, different toilet. Lol. This time there were more cores available, and Node.js clustering had entered the picture. Just when I thought I couldn’t do much better than streaming, I could distribute independent transform workloads across worker processes.
Each worker could stream and transform its own share of the data. Clustering gave us more workers, not some magical ability to divide one stream automatically. We still had to split the jobs sensibly. Once we did, though, the ET part of ETL was on steroids.
The funny part? Eventually my transform processes were waiting for jobs. Not because the transforms were broken, but because the reads and writes couldn’t feed them fast enough. That genuinely took my breath away. I’d spent so much time worrying about processing terabytes that the processors were now waiting on the data. Oh man.
The SD-card express
And that’s only half the tanker story. Every vessel had an Ubuntu server we’d installed, connected to roughly a dozen IoT sensors. We needed to get an enormous amount of data off those ships reliably. Internet access at sea was not exactly cooperating. Ngrok was an absolute pain.
I could wait for a usable ngrok connection and SSH new server-side code onto the vessel. Old-school deployment. Lol. Oddly enough, sending code to the ship was manageable. Downloading terabytes from it was another matter entirely.
So we went gloriously physical. Ship engineers, not software engineers, would plug in an empty terabyte-capacity SD-card device. The Ubuntu server detected an authorised device using public/private-key authentication, then streamed the data directly onto it, encrypted. The encryption was useful, of course, but I admit the spy-movie aspect amused me enormously. What was anyone going to do with terabytes of tanker sensor data anyway?
The server pinged my dashboard back in Singapore. I could monitor the copy and WhatsApp the engineers with progress updates, telling them when to come back. I wanted to expose a little web page so they could see it themselves, but we didn’t have the time. The dashboard and WhatsApp did the job.
When the devices came back to the office, our local machines verified the data. Other engineers then transformed it into the Grafana stack for the data scientists. It was a data-delivery pipeline with a decidedly physical transport layer. Bare metal. No cloud. No nonsense. Fun as hell.
And the bottleneck? The writes! There was something deeply entertaining about building fast streaming transforms only to find that the humble storage device was the one asking everyone else to slow down.
The river still needs floodgates
There’s a difference between filling the reservoir and using its floodgates. The first makes everything wait. The second controls how quickly the water moves. And that’s important, because streaming isn’t about flinging data downstream as fast as possible and hoping for the best. What happens when the destination can’t keep up? We need backpressure: the ability to stop pulling more data until the consumer has capacity. Turn up the input rate, choke the writer, then switch backpressure on and off. Notice what happens to the waiting queue. Backpressure keeps the working queue bounded by slowing down intake when the writer cannot keep up. Illustrative fixed-rate simulation. The buffer holds up to 24 records when backpressure is respected. Without flow control, the model tracks a growing backlog; it does not invent dropped records. Each animation step represents one simulated second.When the destination slows down
Here's the small but important piece of Node that captures it:
for await (const chunk of source) {
if (!destination.write(chunk)) {
await once(destination, 'drain');
}
}
destination.end(); When write() returns false, we wait for drain before consuming more. In production we'd generally prefer pipeline() for its coordination and error handling, but the principle is right there: regulate the flow instead of accumulating an ever-growing queue.
We can also work on data immediately without showing anyone an unfinished result. For example, stream the transformed records into a temporary file, then rename it only when everything succeeds. await pipeline() tells us when that job is done. The processing was happening all along.
And yes, sometimes we have to wait for a complete record. A JSON object might arrive in three chunks. We collect those three chunks, check the record and move on. That's one traveller at immigration, not an entire airport waiting for permission to open.
Back to that image
It's funny that all of this started with watching an image appear in a browser. No grand architecture diagram. No cloud budget. Just a small experiment that made me wonder why we so often wait for things to finish before doing anything useful.
Whether it was bank logs or tanker sensors, the lesson was the same. Let the data move. Check and transform it along the way. Slow things down when the next stage can't keep up. Finishing a job matters, but that doesn't mean we have to wait until the end to start.
I still laugh about the tanker system. We had encrypted SD cards travelling back to Singapore like props from a spy film, and a cluster of transform workers sitting around waiting for the storage devices. It was one helluva ride. And yes, it really did take my breath away.


