Capital Efficiency, Isolation, Risk, and Ruin in Generative AI Inference Serving Infrastructure

Renée St. Amant, Anjaly Parayil, Ankur Mallick, Victor RühlearXiv 2026 · arXiv

This paper from Microsoft researchers argues that the "Information Age" economics of software—where capital intensity was low and idle cycles were cheap—is dead for Generative AI. We have entered a "ruthless economic reality" where inference infrastructure is more akin to heavy industrial machinery. To survive, providers must abandon traditional cloud serving assumptions and embrace workload isolation as a primary lever for capital efficiency.

TL;DR

Generative AI inference costs are dominated by extreme, non-Gaussian request heterogeneity that breaks traditional load balancing and forecasting. By "isolating" similar requests into dedicated fleet partitions, providers can double their capital efficiency, reduce systemic risk, and unlock targeted hardware-software optimizations.

What Problem Does This Paper Address?

The core challenge is capital efficiency: how much "monetizable goodput" (output tokens) can you squeeze out of every dollar spent on GPUs?

In traditional SaaS, workloads are "ergodic"—meaning the system's state can be predicted from past averages. GenAI is fundamentally non-ergodic. A single request might need ten tokens or ten thousand, and its resource footprint (KV cache) grows dynamically.

When you mix these "heavy" and "light" requests in a standard round-robin fleet, the heavy ones act as a "tragedy of the commons," monopolizing memory and compute. This forces the entire fleet to run at lower utilization to maintain latency targets, leading to massive capital waste.

Figure 1. Example of non-Gaussian prompt token distributions (mooncake conversation trace) (Qin et al., 2025)

Methods

The authors propose Workload Isolation: intelligently grouping "like" requests before execution.

Instead of exposing every GPU replica to the full, messy distribution of traffic, the fleet is partitioned. In their primary case study, they partition by input sequence length (Small, Medium, Large).

This transforms a non-ergodic global system into several "pseudo-ergodic" local environments. Once isolated, the "Small" partition can be aggressively optimized for high-concurrency batching, while the "Large" partition can use specialized memory management or speculative decoding without penalizing the rest of the fleet.

Datasets

The study relies on:

  • Production Data: One peak hour of regional production traffic from Microsoft's infrastructure.
  • Mooncake Trace: A publicly available conversation trace (Qin et al., 2025) used to illustrate non-Gaussian token distributions.

Key Insights and Ideas

  • The P95 Trap: In traditional serving, the top 5% of latency events are often ignored as "anomalies." In GenAI, these represent the most resource-intensive (and often most valuable) requests. Ignoring them in planning leads to "ergodic ruin"—systemic crashes or massive over-provisioning.
  • Ergodic Ruin: Without isolation, tail events can permanently exhaust GPU memory or compute budgets, causing "red states" where the system cannot "average out" its performance over time.
  • New Scaling Ratios: Isolation brings visibility. The authors found that long requests are roughly 26x more costly than short ones. In a blended, unpartitioned fleet, this massive differential is invisible, leading to poor investment decisions.
  • Traffic Shaping: By surfacing these costs, engineering teams can intentionally "shape" traffic, incentivizing users toward more capital-efficient request types.
Figure 2. TTFTs for production workload (peak hour) under round robin scheduling. Fitted curve estimates expected TTFT with no request interference or queuing and is hardware dependent. Factor increase over expectation: P50 = 5.9×, P75 = 21.4×, P90 = 33.4×, P95 = 52.9×, P99 = 101×

Key Results

  • 2x Cost Reduction: Simply isolating requests by sequence length resulted in a 2x savings in the number of GPUs needed to serve the same workload.
  • 14x Concurrency Boost: The "Small" request partition achieved an average batch size 14x higher than the "Large" partition while meeting latency targets.
  • Utility per Dollar: The short-sequence partition generated 12x more tokens per hardware instance and processed 24x more requests per instance compared to the large-sequence partition.
Figure 3. Distribution of requests by GPU cost profile for peak hour of production traffic. The mean cost achieved via isolation is 2× below the achieved mean without isolation.

Limitations and Future Work

  • Partition Boundaries: Determining where to draw the lines (e.g., what counts as "Small" vs. "Medium") is a complex, multi-dimensional optimization problem that changes as models and hardware evolve.
  • Stranded Capacity: If partitions are too narrow, you risk having GPUs sit idle waiting for "like" requests. This requires sophisticated architectural choices to maintain provisioning flexibility.
  • Predictive Routing: While input length is known a priori, output length is stochastic. Future work involves using lightweight predictive classifiers to guess request "cost" before routing.
  • Silicon Co-design: The authors suggest that isolation data should inform the design of custom ASICs, moving away from "monolithic, catch-all" architectures toward specialized silicon for different workload segments.

Summary updated

Abstract

Generative AI serving is an extremely capital-intensive endeavor that demands rapid, step-change improvements in capital efficiency to prevent industry wide increases in the cost of capital from limiting growth plans. Keeping pace with the economic necessities will increasingly require infrastructure providers to confront operationally-inconvenient realities of production serving, including the extreme heterogeneity of generative AI request costs. We have explored workload isolation as a foundational, fleet-level lever for increasing capital efficiency. Isolation is the practice of intelligently grouping "like" requests before execution. Rather than exposing every replica to the full heterogeneous workload distribution, isolation deliberately reshapes the workload presented to the hardware by assigning requests with different cost characteristics to different portions of the fleet. Workload isolation on its own does not make expensive requests cheaper; it prevents a costly portion of the workload from unnecessarily determining the operating point and capital efficiency of the entire fleet. It additionally lays the foundational groundwork that can serve as a launching point for compounding gains via narrow-band optimizations (tailoring) and custom silicon designs while also informing up on pricing policy. This paper uses representative production data and experience to convey the motivation behind isolation as a fleet-level lever for increased capital efficiency.
BibTeX
@misc{amant2026capital,
  author = {Renée St. Amant and Anjaly Parayil and Ankur Mallick and Victor Rühle},
  title = {{Capital Efficiency, Isolation, Risk, and Ruin in Generative AI Inference Serving Infrastructure}},
  year = {2026},
  eprint = {2611.08999},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08999},
}
Figure