Capacity Planning for Resource-Constrained Clinical LLM Serving

Yi Tang, Geoffrey Martin, Yiye Zhang, Yifan PengarXiv 2026 · arXiv

TL;DR

When serving LLMs under a fixed hardware budget, how you allocate GPUs to different model sizes (fleet composition) matters far more than the routing algorithm you choose. This study shows that while "all-small" fleets are fastest and "all-large" fleets provide the best long-context quality, balanced heterogeneous fleets (e.g., two small and two large models) offer the most robust operational frontier for diverse clinical workloads.

What Problem Does This Paper Address?

Deploying LLMs in clinical settings often requires self-hosting on dedicated hardware to ensure privacy and data governance. Unlike elastic cloud APIs, these deployments operate within a fixed accelerator budget (e.g., a single 4-GPU node).

Clinical workloads are notoriously heterogeneous: a system might need to handle short, interactive factual queries alongside computationally expensive, long-context hospital-course summarization. This creates a difficult capacity-planning problem: Should you fill your GPUs with many small, fast models or a few large, capable ones? And once that fleet is built, how much can "smart routing" actually improve performance?

Methods

The researchers conducted a controlled study using a four-GPU environment (NVIDIA A100s) and the Qwen2.5 model family (7B as "Small" and 14B as "Large"). They systematically varied three factors across 24 configurations:

  1. Fleet Composition: 4S (all small), 3S+1L, 2S+2L, and 4L (all large).
  2. Routing Policy: Simple RoundRobin, size-aware dispatch based on prompt length (GenericTier), and metadata-based routing (ServiceClass).
  3. Offered Load: Nominal vs. Overloaded regimes.

The system was built on a vLLM-based stack, and the authors measured serving metrics (Time to First Token, latency, throughput), resource utilization, and task-specific quality.

Fig. 1. System diagram for the four-GPU study. The displayed 84/60/96 request counts are nominal counts implied by the intended 0.35/0.25/0.40 mixture, not the realized trace; Table III reports the actual 92/59/89 counts. Routing boxes denote alternative policies, not a sequential cascade. Model-quality labels refer to benchmark coverage proxies, not clinical validation.

Datasets

The study used a "mixed clinical workload" composed of three benchmark-derived task families:

  • PubHealth: Short factual verification (fast, latency-sensitive).
  • BioSSES: Biomedical sentence-pair similarity (moderate reasoning).
  • MIMIC-BHC: Long-context EHR summarization (expensive, quality-sensitive).

Key Insights and Ideas

  • Provisioning Beats Tuning: Fleet composition accounted for the vast majority of variation in performance—52.8% for TTFT and over 65% for latency—while routing policies contributed less than 14%. You cannot "route your way out" of a poorly provisioned fleet.
  • The "Single Large Model" Bottleneck: The 3S+1L configuration (three small, one large) often became a trap. While it improved quality for long-context tasks, the single large instance became a massive bottleneck, leading to queue delays that were significantly worse than even the "all-large" fleet.
  • Balanced Heterogeneity is Robust: The 2S+2L fleet emerged as a strong candidate for general-purpose clinical serving. It avoided the weakest long-context coverage of the 4S fleet while significantly reducing the bottlenecks seen in the 3S+1L setup.
  • Capacity-Aware Gating: In scenarios where large-model capacity is scarce (like 3S+1L), the authors found that "capacity-aware" routing—which tightens the criteria for using the large model when the queue is long—can significantly improve latency and throughput, albeit at a slight cost to task quality.

Key Results

The results highlight a clear trade-off between responsiveness and coverage.

  • Responsiveness: The 4S fleet was the clear winner, with a TTFT of ~0.11s and the highest throughput (~1.7 req/s under overload). However, its MIMIC token recall (a proxy for quality) was the lowest at ~0.19.
  • Quality: The 4L fleet provided the best long-context coverage (recall ~0.26) but was much slower, with TTFT doubling to ~0.21s and p95 latency exceeding 22 seconds.
  • The Frontier: As shown in Figure 2, changing the fleet composition shifts the entire operating regime of the system much more drastically than swapping routing policies.
Primary serving metrics showing how fleet composition (colors) shifts the operating regime more than routing policies (shapes).

Limitations and Future Work

The authors acknowledge several constraints:

  • Model Scope: The study was limited to the Qwen2.5 family. Different model-speed ratios or architectures might shift the "optimal" fleet composition.
  • Proxy Metrics: Quality was measured using automatic proxies (ROUGE, recall) rather than human clinician review, which is the gold standard for safety and factuality.
  • Static Workload: The study used a fixed trace rather than modeling the bursty, unpredictable traffic patterns of a real hospital.
  • Future Directions: Promising areas for follow-up include investigating prefill/decode disaggregation (like DistServe or Splitwise) and more advanced, learned routing policies in these resource-constrained settings.

Summary updated

Abstract

Deploying large language models (LLMs) under a fixed accelerator budget requires jointly deciding how computational capacity is allocated across model sizes and how incoming requests are routed across that capacity. Although prior work has extensively studied LLM routing and serving efficiency, the relative importance of fleet composition and routing under resource constraints remains less understood. We investigate this question using a controlled four-GPU serving environment with heterogeneous clinical NLP workloads spanning short factual verification, biomedical semantic matching, and long-context clinical summarization. We systematically vary fleet composition, routing policy, and offered load while holding the workload trace and hardware budget fixed. Across 24 configurations, descriptive main-effect shares attribute more variation to fleet composition than to the evaluated routing policies in time to first token, latency, and long-context coverage. All-small fleets provide the best responsiveness and throughput but substantially lower long-context coverage, whereas larger-model capacity improves coverage at the cost of serving efficiency. Heterogeneous fleets provide intermediate operating points but can introduce strong-model bottlenecks when large-model capacity is scarce. In a focused evaluation of this bottleneck, capacity-aware routing reduces mean latency and increases throughput while incurring a small reduction in long-context coverage. Within this measured regime, the results support prioritizing capacity allocation, with routing managing demand within the provisioned capacity.
BibTeX
@misc{tang2026capacity,
  author = {Yi Tang and Geoffrey Martin and Yiye Zhang and Yifan Peng},
  title = {{Capacity Planning for Resource-Constrained Clinical LLM Serving}},
  year = {2026},
  eprint = {2611.01152},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01152},
}
Figure