Server Side Dynamic Batching for Accelerated Inference

18 concepts11 papers1 proposal

Hardware accelerators like GPUs and TPUs thrive on parallelism. To maximize throughput, deep learning frameworks traditionally rely on static batching—grouping multiple inputs into a single tensor to process them simultaneously. However, this approach assumes a synchronous world where all inputs arrive at the exact same time and require the exact same amount of computation.

In real-world serving scenarios, this assumption breaks down completely. Requests arrive asynchronously from independent clients, and tasks like Large Language Model (LLM) generation, dynamic neural network execution, or reinforcement learning simulations have highly variable runtimes. If a system uses static batching, it faces a painful trade-off: either wait to accumulate a full batch (introducing massive queueing latency) or run under-filled batches (wasting expensive hardware cycles).

To solve this, a paradigm shift has emerged: server-side dynamic and continuous batching. By moving the batching logic to a centralized server and breaking down the boundaries of a batch from the request level to the iteration level, systems can maximize hardware saturation without sacrificing latency.

Decoupling and Centralized Aggregation

The first step toward optimizing inference is to decouple request generation from model execution. Instead of forcing individual client applications to manage batching, an independent inference server acts as an asynchronous reservoir, pooling requests from multiple sources on the fly.

This architecture is highly effective when integrating modern machine learning into legacy or distributed systems. For instance, in high-energy physics workflows, Michael Wang et al. developed a "GPU-as-a-service" model that intercepts inference requests from single-threaded client processes and aggregates them into optimal batch sizes on a remote server. Similarly, Jeffrey D. Krupa et al. demonstrated that aggregating individual requests from independent client nodes into a single batch at the server level maximizes the utilization of parallel processing units, amortizing the fixed costs of accelerator execution across many concurrent users.

[Client 1] ──(Async Request)──┐
[Client 2] ──(Async Request)──┼─> [Centralized Server Queue] ──> [Optimal Batch] ──> [GPU/TPU]
[Client 3] ──(Async Request)──┘

This decoupling is equally vital for scaling distributed Reinforcement Learning (RL) and web serving pipelines:

  • In Reinforcement Learning: Traditional architectures suffer from "stragglers"—slow environment simulations that hold up the entire batch. To counter this, Lasse Espeholt et al. shifted model inference from decentralized actors to a centralized learner that organizes asynchronous requests into large batches. Taking this further, Dhruv Batra et al. completely decoupled environment workers from inference workers, allowing the system to process action requests as they become available rather than waiting for a fixed batch size.
  • In Retrieval Pipelines: For Retrieval-Augmented Generation (RAG) systems, Eugene Yang et al. showed that server-side asynchronous batching of concurrent HTTP requests significantly enhances GPU utilization, particularly when autonomous agents generate and issue multiple sub-queries simultaneously.

From Batch-Level to Iteration-Level Fluidity

While aggregating asynchronous requests solves the arrival problem, it does not solve the variable execution problem. In autoregressive LLM decoding or recursive networks, some sequences finish much earlier than others. In a static batch, the entire batch must wait for the longest sequence to finish, leaving hardware units idle during the final iterations.

The core breakthrough of modern serving is iteration-level scheduling (or continuous batching). Instead of treating a batch as an immutable block that runs from start to finish, the system re-evaluates the batch at the boundary of every single execution step or token generation iteration.

Static Batching:
Batch 1: [Token 1][Token 2][Token 3][Idle...] -> Waits for longest sequence
Batch 2:                                      [Token 1][Token 2]...

Iteration-Level Batching:
Step 1:  [Req A, Token 1][Req B, Token 1][Req C, Token 1]
Step 2:  [Req A, Token 2][Req B, Finished][Req C, Token 2] -> Req B exits!
Step 3:  [Req A, Token 3][Req D, Token 1][Req C, Token 3] -> Req D enters immediately!

Peizhuang Cong et al. implemented this fluid boundary in their framework, BATON, which allows new queries to occupy processing slots the exact moment an older query completes. This eliminates the idle hardware cycles typically caused by waiting for the longest sequence in a static batch to finish.

This concept applies across different granularities of neural network execution:

  • Queueing Theory: Yuedong Xu et al. analyzed this phenomenon mathematically. By modeling inference as an M/G/1 queue, the authors proved that "elastic batching"—releasing finished requests immediately—minimizes intra-batch waiting and reduces the lower bound of queueing delay across varying token distributions.
  • Recursive Architectures: For models that reuse layers dynamically, Sangmin Bae et al. introduced continuous depth-wise batching. In their scheduling mechanism, tokens exit the computation loop independently, allowing new requests to fill the vacated processing slots immediately and process different samples at different iteration stages within the same batch.
  • Dynamic Networks: Even when network structures change per input, Shizhen Xu et al. demonstrated that separating static computation logic from dynamic input graphs allows the runtime to group computations for efficient parallel processing.

Amortizing Costs and Overcoming Remoting Overhead

Why does dynamic orchestration yield such massive performance gains? It fundamentally changes how hardware costs are amortizing during complex tasks.

For instance, when running large Mixture-of-Experts (MoE) models, transferring weights or activations creates severe bottlenecks. Keisuke Kamahori et al. discovered that dynamic orchestration strategies yield the highest performance gains during tasks like beam search, which naturally increase the effective batch size. This allows the system to more effectively leverage the high-throughput capabilities of the GPU by amortizing the cost of weight transfers across multiple parallel sequences.

However, moving batching to the server side introduces a new challenge: network and API overhead. If a client has to wait synchronously for a remote GPU call to return, the communication latency can destroy the throughput gains.

To resolve this bottleneck, Minchen Yu et al. proposed asynchronously redirecting CUDA API calls within serverless GPU computing. By allowing inference functions to issue commands to the remote GPU without waiting for immediate responses, the system hides the remoting overhead. This ensures that the centralized server can focus entirely on what it does best: packing, scheduling, and executing dense batches at peak hardware efficiency.

Key Foundational Work

GPU-accelerated machine learning inference as a service for computing in neutrino experiments (2020) — contributed 2 concept elements to this direction.
GPU coprocessors as a service for deep learning inference in high energy physics (2020) — contributed 2 concept elements to this direction.
SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_server_side_dynamic_batching_22f81e21,
  title = {server-side dynamic batching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/server-side-dynamic-batching/rdr_22f81e21f79c4a3caf2e8522c2190413},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_server_side_dynamic_batching_22f81e21_v1,
  title = {server-side dynamic batching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/server-side-dynamic-batching/rdr_22f81e21f79c4a3caf2e8522c2190413/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake