When designing systems to process data streams, engineers encounter a fundamental tension: should the system maximize the total volume of data processed over time (throughput), or minimize the time it takes to process any single piece of data (latency)?
In an ideal world, a system would achieve both peak throughput and ultra-low latency. In practice, however, these two goals directly undermine each other. As explored by Aashwin Ananda Mishra et al. in Testing the data framework for an AI algorithm in preparation for high data rate X-ray facilities and Panji Setiawan et al. in Revisiting some model-based and data-driven denoising algorithms in Aurora 2 context, choosing between streaming data record-by-record or aggregating it into batches forces a stark architectural compromise.
To understand why this trade-off exists, we have to look at how modern hardware and software manage the overhead of computation.
Why High Throughput Demands Batching
To maximize throughput, general-purpose computing architectures—such as CPUs, GPUs, and distributed clusters—rely on a simple strategy: amortize fixed overhead by aggregating work.
Every time a processor executes a task, it incurs non-computational costs. It must fetch instructions, load model weights or data from slow main memory into fast local caches, and synchronize parallel threads or distributed nodes. If a system processes a single data point at a time, these fixed overhead costs are paid for every single sample. The processor spends more time waiting for data to move than it does performing actual computation.
Batching solves this by grouping multiple inputs together. For example, in distributed frameworks like MapReduce, processing massive datasets in aggregate allows the system to maximize overall data rates, though individual task completion times are sacrificed (Jimmy Lin et al. in Data Intensive Text Processing with MapReduce). Similarly, GPUs achieve massive parallel efficiency during offline neural network training by sharing the cost of loading large weight matrices across hundreds of inputs simultaneously (Hongxiang Fan et al. in Toward Full-Stack Acceleration of Deep Convolutional Neural Networks on FPGAs).
The Reality of Real-Time Streams: Batch Size One
While batching is highly efficient for offline processing, it fails in real-time edge scenarios. In mission-critical applications like autonomous driving, medical robotics (Weiwen Jiang et al. in Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference), or particle accelerator control (Seda Ogrenci et al. in ML-based Real-Time Control at the Edge: An Approach Using hls4ml), a system cannot afford to wait for a batch of inputs to accumulate. It must respond immediately to individual data points as they arrive in a continuous stream.
This operational constraint forces a batch size of one. When forced into this sequential, single-sample mode, the parallel efficiency of general-purpose architectures collapses:
- Resource Management Overhead: High-performance processors struggle because the administrative overhead of managing parallel resources and scheduling threads outweighs the actual computation of a single sample (Shih-Chii Liu et al. in EdgeDRNN: Enabling Low-latency Recurrent Neural Network Edge Inference).
- Memory Bottlenecks: Without a batch of inputs to reuse the same weights in cache, the system must repeatedly fetch model parameters from main memory for every individual input, exposing severe memory bus bottlenecks (Chaim Baskin et al. in Streaming Architecture for Large-Scale Quantized Neural Networks on an FPGA-Based Dataflow Platform).
The performance penalty of this reality is severe. As demonstrated by the industry-standard benchmarks evaluated by Vijay Janapa Reddi et al. in MLPerf Inference Benchmark, server-side environments operating under strict real-time latency budgets can lose more than half of their peak potential throughput simply because they cannot effectively batch incoming queries.
Batch Processing (High Throughput, High Latency)
[Input 1] ──┐
[Input 2] ──┼─► [ Accumulate Batch ] ──► [ High-Parallel Compute ] ──► [ Outputs 1-3 ]
[Input 3] ──┘
Streaming / Batch-Size-One (Low Throughput, Low Latency)
[Input 1] ─────────────────────────────► [ Exposed Overhead Compute ] ──► [ Output 1 ]
The Batching Paradox in Software and Networks
This conflict is not confined to hardware execution units; it spans the entire software and networking stack.
Standard commodity servers and networking frameworks are traditionally optimized to maximize data rates by buffering packets. However, as noted by Charalampos Stylianopoulos et al. in On the performance of commodity hardware for low latency and low jitter packet processing, packet buffering and queuing introduce significant delays and timing variability (jitter) that are unacceptable for real-time industrial applications.
This tension manifests as the batching paradox in distributed stream processing frameworks. As explained by Song Wu et al. in TurboStream: Towards Low-Latency Data Stream Processing, communication frameworks queue and batch messages to maximize network throughput, but this optimization inherently injects per-message queuing latency.
Similarly, output buffers are essential for minimizing data transfer overhead between streaming operators, yet they serve as a primary driver of latency (Björn Lohrmann et al. in Nephele streaming: stream processing under QoS constraints at scale). Tuning these buffers creates a delicate operational balance: shrinking buffer sizes reduces per-event latency, but it can severely degrade the system's capacity to handle high data volumes or support multiple concurrent client applications, as observed in the computer vision pipeline designed by the authors of Low Cost, High Performance Automatic Motorcycle Helmet Violation Detection.
Architectural Strategies to Break the Trade-off
To bypass the throughput-latency trade-off, researchers are developing systems that move away from traditional batch-optimized paradigms.
1. Dual-Paradigm Processing
Rather than forcing a permanent choice between responsiveness and raw processing power, some modern software frameworks adapt dynamically. For instance, the framework proposed by Song Wu et al. in Dual-Paradigm Stream Processing dynamically switches between record-by-record streaming during normal operation to maintain low latency, and micro-batching during sudden data bursts to sustain high throughput.
2. Custom Streaming Hardware
Instead of relying on general-purpose processors that require batching to hide memory latency, edge applications increasingly utilize specialized hardware like Field-Programmable Gate Arrays (FPGAs). FPGAs allow engineers to design custom, deeply pipelined streaming dataflow architectures (Chaim Baskin et al. in Streaming Architecture for Large-Scale Quantized Neural Networks on an FPGA-Based Dataflow Platform).
Because these custom architectures can store network weights directly in on-chip registers and pass data sequentially from layer to layer without hitting an external memory bus, they achieve near-perfect compute efficiency even at a batch size of one (Hongxiang Fan et al. in Toward Full-Stack Acceleration of Deep Convolutional Neural Networks on FPGAs). Furthermore, by offloading data traffic from the slow main memory bus to dedicated inter-chip links, frameworks like the one designed by Weiwen Jiang et al. in Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference can scale real-time inference across multiple FPGAs, slashing single-task latency without relying on parallel data streams.
Key Foundational Work
Revisiting some model-based and data-driven denoising algorithms in Aurora 2 context (2004) — contributed 2 concept elements to this direction.
Data Intensive Text Processing with MapReduce (2009) — contributed 1 concept elements to this direction.