Compositional Conformal Risk Control for Multi-Stage LLM Cascades

Yuxia Sun, Jiahuang Wang, Haoting Liu, Zhiwei Huang, Tingrui Pei, Saiqin LongarXiv 2026 · arXiv

TL;DR

CompCRC (Compositional Conformal Risk Control) is a new framework for managing multi-stage LLM cascades that provides a rigorous statistical guarantee on error rates. By decomposing a global risk budget into individual stages, it allows developers to optimize for cost and accuracy while providing fine-grained diagnostic information about which model in the chain is failing to meet reliability standards.

What Problem Does This Paper Address?

Deploying Large Language Models (LLMs) at scale is a constant tug-of-war between accuracy and inference cost. A common solution is "cascading": start with a small, cheap model (like Llama-3-3B) and only escalate to a larger, expensive model (like Qwen-14B) if the first model is unsure.

However, these cascades are often "black boxes" regarding reliability. While we can measure their average performance, it is difficult to provide a formal guarantee that the system won't exceed a specific error rate (e.g., "no more than 10% of accepted answers should be wrong"). Furthermore, if a cascade does fail, existing methods can't tell you which stage was the culprit. Was the small model too confident, or did the large model fail on the difficult edge cases?

Methods

The authors introduce CompCRC, which applies the principles of Conformal Risk Control (CRC) to multi-stage cascades. The core innovation is the additive decomposition of risk.

Since a query in a cascade is accepted by exactly one stage (or falls through to a final fallback), the total system risk is simply the sum of the risks contributed by each stage. This allows the authors to:

  1. Allocate Budgets: Split a global risk budget ($\alpha$) into local budgets ($\alpha_k$) for each stage.
  2. Construct Candidates: Generate various cascade policies using different weighting rules (e.g., favoring cheaper models or models with lower error rates).
  3. Select & Certify: Use independent data to select the most cost-efficient policy and then "certify" it. A stage is certified only if its observed error rate (with a small statistical correction) stays within its local budget.
Figure 1: Overview of CompCRC. Candidate policies are fixed before independent selection and stage-wise certification. Here, 𝐾denotes the number of cascade stages, 𝑚𝑘counts incorrect answers accepted at stage 𝑘on 𝐷𝑘, and 𝑛𝑘= |𝐷𝑘|. The selected cascade is enabled only if all stages pass certification. An empty candidate set or certification failure triggers the predefined fallback. For an enabled cascade, cascade-end fallback returns the highest-confidence answer among the already generated outputs when all stages reject a query.

The deployment is "gated": the cascade is only enabled if every single stage passes its local certification. If any stage fails, the system reverts to a safe, predefined fallback policy.

Datasets

The framework was evaluated using a three-stage cascade (Llama-3.2-3B $\rightarrow$ GLM-4-9B $\rightarrow$ Qwen3-14B) across three diverse benchmarks:

  • MMLU: General knowledge and multi-task language understanding.
  • GSM8K: Multi-step mathematical reasoning.
  • HellaSwag: Commonsense reasoning.

Key Insights and Ideas

  • Risk as a Finite Resource: By treating risk as a budget to be spent across stages, CompCRC turns a fuzzy reliability problem into a clear allocation problem.
  • Cost-Efficiency Index (CEI): The authors propose a metric ($\text{AvgCost} / \text{Accuracy}$) to pick the best candidate policy. This ensures the system isn't just "safe," but also economically viable.
  • Diagnostic Power: Unlike "Global-CRC," which only gives a thumbs-up or thumbs-down to the whole system, CompCRC identifies exactly which stage is violating its budget. For example, in their MMLU experiments at a tight risk budget ($\alpha=0.30$), the system correctly identified that the first two stages were the sources of certification failure.
  • Flexible Allocation Rules: The authors tested eight different rules for splitting the budget. They found that "Cost-based" allocation—giving more risk budget to cheaper models—significantly improved the accuracy-per-unit-cost.
The eight candidate risk-budget allocation rules used to weight different stages.

Key Results

CompCRC consistently outperformed standard global risk control methods:

  • Lower Risk, Higher Accuracy: At a global budget of $\alpha=0.45$, CompCRC reduced observed risk by up to 2.0 percentage points and improved final accuracy by up to 2.0 percentage points compared to Global-CRC.
  • Massive Cost Savings: On the HellaSwag benchmark, CompCRC reduced weighted inference costs by 30.37% compared to the global baseline.
  • Better Efficiency: The cost-based budget allocation improved accuracy-per-unit-cost by 10.6% to 46.7% over a simple uniform allocation.

Limitations and Future Work

While CompCRC provides strong theoretical guarantees, the authors note several areas for improvement:

  • Data Requirements: The method requires partitioning data into multiple sets (construction, selection, and certification), which might be challenging for data-scarce tasks.
  • Static Environments: The current proof relies on "exchangeability," meaning it assumes the data distribution doesn't change over time. Future work is needed to handle "distribution shift" (e.g., when user queries change style or topic).
  • Latency vs. API Cost: The current cost model focuses on API pricing/token counts. Incorporating actual wall-clock latency would make the framework even more practical for real-time engineering.
  • Fallback Risk: The current guarantee only covers errors from accepted answers. Extending this to cover errors that occur in the final fallback stage is a logical next step.

Summary updated

Abstract

Multi-stage large language model (LLM) cascades balance accuracy and inference cost, but reliable deployment also requires controlling error risk and identifying its sources across stages. We propose Compositional Conformal Risk Control (CompCRC), a risk-control framework for pre-specified multi-stage LLM cascades. We decompose the risk of accepting incorrect answers into additive stage-wise contributions over the full query population, linking local risk budgets to a system-level guarantee. CompCRC constructs and freezes candidate budget allocations and acceptance thresholds, selects one candidate on independent data, and certifies its stage-wise risk contributions on separate held-out data. Under conditional exchangeability, we establish a finite-sample guarantee: the marginal expected accepted-error risk of the complete deployment procedure does not exceed the global risk budget. At the reported comparison budget of $α=0.45$, all three stages of CompCRC pass their local checks on MMLU, GSM8K, and HellaSwag. Compared with Global-CRC, which passes system-level risk certification at the same budget, CompCRC reduces observed accepted-error risk by 1.33--2.00 percentage points, improves final accuracy by 0.67--2.00 percentage points, and reduces weighted inference cost by 30.37\% on HellaSwag. The selected cost-based budget allocation also improves accuracy per unit cost by 10.6\%--46.7\% over uniform allocation. Further experiments show that stage-wise statistics identify stages that fail local certification checks, providing diagnostic information alongside system-level risk control.
BibTeX
@misc{sun2026compositional,
  author = {Yuxia Sun and Jiahuang Wang and Haoting Liu and Zhiwei Huang and Tingrui Pei and Saiqin Long},
  title = {{Compositional Conformal Risk Control for Multi-Stage LLM Cascades}},
  year = {2026},
  eprint = {2611.11114},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11114},
}
Figure