TL;DR
When serving LLMs under a fixed hardware budget, how you allocate GPUs to different model sizes (fleet composition) matters far more than the routing algorithm you choose. This study shows that while "all-small" fleets are fastest and "all-large" fleets provide the best long-context quality, balanced heterogeneous fleets (e.g., two small and two large models) offer the most robust operational frontier for diverse clinical workloads.
What Problem Does This Paper Address?
Deploying LLMs in clinical settings often requires self-hosting on dedicated hardware to ensure privacy and data governance. Unlike elastic cloud APIs, these deployments operate within a fixed accelerator budget (e.g., a single 4-GPU node).
Clinical workloads are notoriously heterogeneous: a system might need to handle short, interactive factual queries alongside computationally expensive, long-context hospital-course summarization. This creates a difficult capacity-planning problem: Should you fill your GPUs with many small, fast models or a few large, capable ones? And once that fleet is built, how much can "smart routing" actually improve performance?
Methods
The researchers conducted a controlled study using a four-GPU environment (NVIDIA A100s) and the Qwen2.5 model family (7B as "Small" and 14B as "Large"). They systematically varied three factors across 24 configurations:
- Fleet Composition: 4S (all small), 3S+1L, 2S+2L, and 4L (all large).
- Routing Policy: Simple RoundRobin, size-aware dispatch based on prompt length (GenericTier), and metadata-based routing (ServiceClass).
- Offered Load: Nominal vs. Overloaded regimes.
The system was built on a vLLM-based stack, and the authors measured serving metrics (Time to First Token, latency, throughput), resource utilization, and task-specific quality.
Datasets
The study used a "mixed clinical workload" composed of three benchmark-derived task families:
- PubHealth: Short factual verification (fast, latency-sensitive).
- BioSSES: Biomedical sentence-pair similarity (moderate reasoning).
- MIMIC-BHC: Long-context EHR summarization (expensive, quality-sensitive).
Key Insights and Ideas
- Provisioning Beats Tuning: Fleet composition accounted for the vast majority of variation in performance—52.8% for TTFT and over 65% for latency—while routing policies contributed less than 14%. You cannot "route your way out" of a poorly provisioned fleet.
- The "Single Large Model" Bottleneck: The 3S+1L configuration (three small, one large) often became a trap. While it improved quality for long-context tasks, the single large instance became a massive bottleneck, leading to queue delays that were significantly worse than even the "all-large" fleet.
- Balanced Heterogeneity is Robust: The 2S+2L fleet emerged as a strong candidate for general-purpose clinical serving. It avoided the weakest long-context coverage of the 4S fleet while significantly reducing the bottlenecks seen in the 3S+1L setup.
- Capacity-Aware Gating: In scenarios where large-model capacity is scarce (like 3S+1L), the authors found that "capacity-aware" routing—which tightens the criteria for using the large model when the queue is long—can significantly improve latency and throughput, albeit at a slight cost to task quality.
Key Results
The results highlight a clear trade-off between responsiveness and coverage.
- Responsiveness: The 4S fleet was the clear winner, with a TTFT of ~0.11s and the highest throughput (~1.7 req/s under overload). However, its MIMIC token recall (a proxy for quality) was the lowest at ~0.19.
- Quality: The 4L fleet provided the best long-context coverage (recall ~0.26) but was much slower, with TTFT doubling to ~0.21s and p95 latency exceeding 22 seconds.
- The Frontier: As shown in Figure 2, changing the fleet composition shifts the entire operating regime of the system much more drastically than swapping routing policies.
Limitations and Future Work
The authors acknowledge several constraints:
- Model Scope: The study was limited to the Qwen2.5 family. Different model-speed ratios or architectures might shift the "optimal" fleet composition.
- Proxy Metrics: Quality was measured using automatic proxies (ROUGE, recall) rather than human clinician review, which is the gold standard for safety and factuality.
- Static Workload: The study used a fixed trace rather than modeling the bursty, unpredictable traffic patterns of a real hospital.
- Future Directions: Promising areas for follow-up include investigating prefill/decode disaggregation (like DistServe or Splitwise) and more advanced, learned routing policies in these resource-constrained settings.
Summary updated