Introduction
When users prompt a Large Language Model (LLM), the provider (such as OpenAI or Anthropic) often decides behind the scenes whether to send the query to a small, fast model or a large, expensive reasoning model. This decision process is known as routing. If the smaller model fails to provide a satisfactory answer, the system might automatically escalate the query to the larger model—a process called cascading.
While extensive research focuses on making these routing systems more accurate or cost-effective, most of it treats the user as a passive participant. In reality, users are active agents: if a model is too slow or repeatedly gives poor answers, a user might re-prompt, or they might simply abandon the task and cancel their subscription. This paper addresses the "welfare gap"—the fundamental misalignment between a routing strategy designed to save the provider money and one that actually satisfies the user.
A Game-Theoretic Framework
The authors formalize the interaction between the LLM provider and the user as a Stackelberg Game (a leader-follower game).
- The Provider (Leader): Sets a routing policy (which model to start with) and a cascading rule (the probability of escalating to a better model if the first one fails). Their objective is to minimize the combined cost of compute and the penalty of losing a subscriber.
- The User (Follower): Observes the system's behavior and decides on an "abandonment policy"—essentially, how many times they are willing to re-prompt before quitting. Their goal is to maximize the value of a successful answer minus the "cost" of the time spent waiting (latency).
To capture these dynamics, the researchers utilize a Markov Model to track the state of the interaction, deriving closed-form mathematical expressions to find the equilibrium where both parties act in their own best interest.
Key Insights: Routing, Cascades, and Throttling
By analyzing the mathematical boundaries of this game, the authors uncover several counterintuitive insights about how providers and users interact.
Net Value is King User behavior is primarily driven by "net value" (Expected Value minus Latency). If the delay of using a model outweighs the benefit of the answer, the user will quit immediately, regardless of how capable or "smart" the model is.
Cascades are Overrated Surprisingly, the authors found that in almost all cases, the optimal strategy for a provider is a static route (picking one model and sticking to it) rather than a complex cascade. Cascading only proves beneficial in a very narrow mathematical window where models are perfectly differentiated in their value and delay.
The Misalignment Gap There is a fundamental conflict when the provider’s cheapest model is not the user’s preferred choice. If the financial penalty for a user quitting is low, the provider will choose the route that saves them the most compute money, even if it significantly degrades the user's experience.
The Incentive to Throttle Perhaps the most cynical finding is that if users are "sticky" (highly unlikely to unsubscribe), providers are mathematically incentivized to throttle latency. By intentionally making the models slower, providers discourage users from engaging in "expensive" repeated interactions. In this scenario, delay acts as a hidden price increase used to protect the provider's margins at the user's expense.
Results and Implications
The paper yields simple, actionable threshold rules that providers can use to determine the best model to route to based on a user's likely patience. Furthermore, the authors fully characterize the "User Best Response," showing exactly when a user switches from being patient to impatient based on the provider's routing choices. Finally, they provide mathematical proof demonstrating that when the cost of losing a user is lower than the cost of compute, throttling latency becomes the dominant strategy for the provider to minimize service costs.
Limitations and Future Directions
While this theoretical framework provides strong foundational insights, it relies on a few simplifying assumptions:
- Two-Model Simplification: The study focuses on a binary world (Standard vs. Reasoning), whereas real-world providers manage dozens of model variants.
- Static Strategies: The model assumes the user decides on a fixed abandonment strategy upfront, rather than getting progressively more annoyed with each failed prompt (dynamic abandonment).
- Pricing Structures: The paper assumes a fixed-fee subscription model. Dynamics might shift significantly under "pay-as-you-go" API pricing where users pay per token.
- Information Symmetry: The authors assume users can perfectly observe the provider's routing and success rates. Future work could explore "hidden" routing where the user has to guess which model is serving their query.
Summary updated