# Promises vs Streams: Dams, Rivers and Immigration Checkpoints

Canonical URL: https://geekist.co/promises-vs-streams-dams-rivers-and-immigration-checkpoints/
Published (UTC): 2026-11-07T06:43:52Z
Updated (UTC): 2026-11-15T11:23:57Z
Author: GuruGeek
Categories: Tame the Tech, Beneath the Stack

*Or how an image loading in a browser eventually led to five terabytes a day, oil tankers, and a rather suspicious collection of encrypted SD cards.*

I can’t remember exactly when I first encountered Node.js. It was early enough that asynchronous programming frequently meant nesting callbacks until the code resembled an archaeological excavation site. I wasn’t using Node professionally. I was just playing with it, and one experiment involved streaming an image to a browser.

The image appeared progressively, from top to bottom, as the bytes arrived. I was used to applications that waited and then presented the finished thing. Here was a browser doing useful work before the operation had finished. I was blown away!

Before anyone rushes to explain that Apache could stream too: yes, we know. What fascinated me wasn’t a server capability. It was the way I’d been thinking about work. Why did I keep treating *finished* as a prerequisite for *started*?

I began organising Node applications around events, with named handlers I could actually follow rather than callbacks nesting into the Earth’s crust. I didn’t know anything about functional programming at the time. I just liked what happened when the data was allowed to move.

## You’re building a bloody dam again!

This eventually became a phrase I’d throw at fellow engineers: *“You’re building a bloody dam again!”* Usually when somebody read an entire payload into memory just to do something that could have been done as it arrived. Lol.

Think of a river. If we insist on filling a reservoir before using a single drop, we’ve turned a continuous flow into a batch. Sometimes collecting everything first is necessary. But if the job is to inspect, filter or transform what passes by, why make the whole river wait? That’s the bloody dam I’m complaining about.

![A vast reservoir is trapped behind a solid concrete dam while the riverbed downstream is nearly dry.](https://img.geekist.co/wp-content/uploads/2026/11/node-the-bloody-dam-full-reservoir-1024x512.avif)

A reservoir full of data. Nothing useful moves downstream until the entire payload has arrived.

01

SOURCE

Incoming data

→

02

RESERVOIR

Buffer entire input

→

03

PROCESS

Only after EOF

→

04

RELEASE

All at once

Swipe to explore the flow →

Flow architecture**The full reservoir**Nothing useful leaves until the entire payload arrives

INTERACTIVE FIELD NOTES · 01

### Follow the data

Watch eight complete records move through the same pipeline. The difference is *when* we allow work to begin.

Process as it arrives

Wait for the whole payload

01 · INTAKE**Read****0**/8

→

02 · RECORD**Validate****0**/8

→

03 · MAP**Transform****0**/8

→

04 · OUTPUT**Write****0**/8

A complete record can be processed without waiting for the rest of the payload.

↻ Replay

Pause

Conceptual illustration. We are comparing an explicit buffering decision with incremental record processing; a Promise does not itself buffer data.

Now picture airport immigration. Travellers arrive, an officer checks each passport, and each traveller moves on. Nobody waits for every flight of the day to land before opening the first passport. That’s the useful checkpoint: validate one complete record, then let it through. If the counter can’t keep up, the queue slows the arrivals. If there’s more independent work than one counter can handle, we can open another.

![Travellers pass through an airport immigration checkpoint individually, with a passport being inspected while cleared passengers continue onwards.](https://img.geekist.co/wp-content/uploads/2026/11/node-immigration-incremental-checkpoint-1024x512.avif)

One traveller at a time. Validate the complete record, let it through, and keep the queue moving.

01

SOURCE

Records arrive

→

02

INSPECT

Validate one record

→

03

TRANSFORM

Do useful work

→

04

CONTINUE

Next record

Swipe to explore the flow →

Flow architecture**The immigration counter**Each complete record can be checked and passed onwards

That’s really the difference. We can wait for everything to arrive, or start working as it comes in. Promises aren’t the villain here. They tell us when a job has finished. Streams let us work while it’s still happening. Quite often, we want both.

Early Node made that distinction wonderfully visible. Its HTTP response API let us `write()` repeatedly and `end()` when we were done. Yet a familiar pattern in early web tooling was to collect all the `data` events into a buffer and hand it to a callback only at `end`. Here’s the shape of it, from [Express 0.10.0’s HTTP client](https://github.com/expressjs/express/blob/0.10.0/lib/express/http.js):

``` EnlighterJSRAW
res.on('data', function (chunk) {
  buf += chunk;
});

res.on('end', function () {
  callback(null, buf, res);
});
```

Sometimes we really do need the complete response. Fair enough. But here the decision has already been made for us. Nobody gets to touch the data until the last chunk arrives. A perfectly good stream has become a waiting room.

## Five terabytes. One M2.

It really, really, really shines with transform streams. I once had to turn terabytes of JSON logs into CSV for the BI team at DBS, while working on its [marketplace platform](https://www.dbs.com.sg/personal/marketplace/index.html). Around **5TB a day**. The official route involved a military hierarchy of requests to commission high-end AWS machines. By the time those approvals arrived, I’d have missed the deadline.

So I tried it on an M2. Believe it or not, I was sceptical myself. Then it worked. Not only did it work, the bloody thing was practically flying.

The trick wasn’t some exotic algorithm. It was refusing to treat five terabytes as one object. Read the input incrementally, identify complete records, validate and transform them, and keep moving. The output could branch: CSV for the BI team on one side, compressed archives for S3 on the other. No need to hold the whole dataset hostage while waiting for the last byte.

01

READ

JSON log stream

→

02

FRAME

Complete records

→

03

BRANCH

Coordinate consumers

→

↗   ↘

04

OUTPUT A

CSV for BI

05

OUTPUT B

Compressed S3 archive

Swipe to explore the flow →

Flow architecture**Five terabytes, one pass**Independent outputs without a five-terabyte working set

Of course, splitting a stream into two destinations needs a little care. What if S3 slows down while the CSV writer races ahead? Both branches need their own flow control and error handling. Get that right, though, and we can produce both outputs in one pass without swallowing the entire dataset first.

When I tell other engineers this story, I often get a suspicious look. Five terabytes? On that machine? I understand. *I* didn’t quite believe it until I watched it happen. We’re so accustomed to equating large datasets with large fleets of hardware that we forget the distinction between the size of the *whole* dataset and the size of the *working set*.

## Then someone gave me more cores

Later, at [Chord X](https://imcventures.vc/2021/10/15/imc-industrial-group-and-imc-ventures-to-test-chord-x-solutions-on-maritime-tranquility/), I had to process terabytes of IoT data from oil tankers. Same shit, different toilet. Lol. This time there were more cores available, and Node.js clustering had entered the picture. Just when I thought I couldn’t do much better than streaming, I could distribute independent transform workloads across worker processes.

Each worker could stream and transform its own share of the data. Clustering gave us more workers, not some magical ability to divide one stream automatically. We still had to split the jobs sensibly. Once we did, though, the ET part of ETL was on steroids.

01

JOBS

Partition input

→

02

WORKER A

Transform

03

WORKER B

Transform

04

WORKER C

Transform

→

05

RESULT

Verified outputs

Swipe to explore the flow →

Flow architecture**More counters, not a bigger waiting room**Partition jobs, then run independent transforms across workers

The funny part? Eventually my transform processes were waiting for jobs. Not because the transforms were broken, but because the reads and writes couldn’t feed them fast enough. That genuinely took my breath away. I’d spent so much time worrying about processing terabytes that the processors were now waiting on the data. Oh man.

## The SD-card express

And that’s only half the tanker story. Every vessel had an Ubuntu server we’d installed, connected to roughly a dozen IoT sensors. We needed to get an enormous amount of data off those ships reliably. Internet access at sea was not exactly cooperating. Ngrok was an absolute pain.

I could wait for a usable ngrok connection and SSH new server-side code onto the vessel. Old-school deployment. Lol. Oddly enough, sending code *to* the ship was manageable. Downloading terabytes *from* it was another matter entirely.

So we went gloriously physical. Ship engineers, not software engineers, would plug in an empty terabyte-capacity SD-card device. The Ubuntu server detected an authorised device using public/private-key authentication, then streamed the data directly onto it, encrypted. The encryption was useful, of course, but I admit the spy-movie aspect amused me enormously. What was anyone going to do with terabytes of tanker sensor data anyway?

The server pinged my dashboard back in Singapore. I could monitor the copy and WhatsApp the engineers with progress updates, telling them when to come back. I wanted to expose a little web page so they could see it themselves, but we didn’t have the time. The dashboard and WhatsApp did the job.

01

AT SEA

Sensors → Ubuntu

→

02

AUTHENTICATE

Authorised device

→

03

STREAM

Encrypted SD card

→

04

ON LAND

Office verification

→

05

ANALYSE

Grafana pipeline

Swipe to explore the flow →

Flow architecture**The SD-card express**The physical transport layer was the network

When the devices came back to the office, our local machines verified the data. Other engineers then transformed it into the Grafana stack for the data scientists. It was a data-delivery pipeline with a decidedly physical transport layer. Bare metal. No cloud. No nonsense. Fun as hell.

And the bottleneck? The writes! There was something deeply entertaining about building fast streaming transforms only to find that the humble storage device was the one asking everyone else to slow down.

## The river still needs floodgates

There’s a difference between filling the reservoir and using its floodgates. The first makes everything wait. The second controls how quickly the water moves. And that’s important, because streaming isn’t about flinging data downstream as fast as possible and hoping for the best. What happens when the destination can’t keep up? We need *backpressure*: the ability to stop pulling more data until the consumer has capacity.

INTERACTIVE FIELD NOTES · 02

### When the destination slows down

Turn up the input rate, choke the writer, then switch backpressure on and off. Notice what happens to the waiting queue.

Incoming records

14/s

2/s20/s

Writer capacity

6/s

2/s20/s

Respect backpressure?

On

Off

INPUTBUFFERWRITER

Waiting records****0** queued**

Accepted / sec**14**

Written / sec**6**

Source held back**0**

Backpressure keeps the working queue bounded by slowing down intake when the writer cannot keep up.

Simulation: 0 of 12 seconds

↻ Replay

Pause

Illustrative fixed-rate simulation. The buffer holds up to 24 records when backpressure is respected. Without flow control, the model tracks a growing backlog; it does not invent dropped records. Each animation step represents one simulated second.

Here's the small but important piece of Node that captures it:

``` EnlighterJSRAW
for await (const chunk of source) {
  if (!destination.write(chunk)) {
    await once(destination, 'drain');
  }
}
destination.end();
```

When `write()` returns `false`, we wait for `drain` before consuming more. In production we'd generally prefer `pipeline()` for its coordination and error handling, but the principle is right there: regulate the flow instead of accumulating an ever-growing queue.

We can also work on data immediately without showing anyone an unfinished result. For example, stream the transformed records into a temporary file, then rename it only when everything succeeds. `await pipeline()` tells us when that job is done. The processing was happening all along.

And yes, sometimes we have to wait for a complete record. A JSON object might arrive in three chunks. We collect those three chunks, check the record and move on. That's one traveller at immigration, not an entire airport waiting for permission to open.

## Back to that image

It's funny that all of this started with watching an image appear in a browser. No grand architecture diagram. No cloud budget. Just a small experiment that made me wonder why we so often wait for things to finish before doing anything useful.

Whether it was bank logs or tanker sensors, the lesson was the same. Let the data move. Check and transform it along the way. Slow things down when the next stage can't keep up. Finishing a job matters, but that doesn't mean we have to wait until the end to start.

I still laugh about the tanker system. We had encrypted SD cards travelling back to Singapore like props from a spy film, and a cluster of transform workers sitting around waiting for the storage devices. It was one helluva ride. And yes, it really did take my breath away.

