Compound AI System Reliability: A Failure Taxonomy and Resilience Pattern Catalog from 150 Production Incidents

Rudrendu Kumar Paul, Sourav NandyAIWILD @ ICML 2026 · OpenReview

TL;DR

This paper shifts the focus of AI reliability from individual models to "compound systems" (like RAG and agents), identifying 23 specific failure modes that occur at the boundaries between components. By analyzing 150 real-world production incidents and testing five "resilience patterns," the authors demonstrate that combining these patterns can reduce mean-time-to-recovery (MTTR) by a staggering 71%.

What Problem Does This Paper Address?

We’ve become quite good at measuring if a single LLM is "hallucinating," but we are surprisingly bad at monitoring Compound AI Systems—architectures that glue together retrievers, generators, tools, and orchestrators.

The authors highlight a "tooling gap": standard observability platforms track metrics like latency and token counts per component, but they miss inter-component semantic failures. For example, a retriever might successfully return documents (passing its health check), but if those documents are irrelevant due to an index drift, the generator will confidently produce a wrong answer. To the monitoring system, everything looks "green," while the user receives garbage. This "silent quality degradation" is the primary challenge the paper tackles.

Methods

The researchers employed a two-phase approach:

  1. Taxonomy Construction: They analyzed 150 production incident reports (97 from open-source projects like LangChain and LlamaIndex, and 53 from anonymized enterprise deployments). They used "open coding" to categorize these incidents into a taxonomy of 23 failure modes across five categories.
  2. Fault Injection Experiments: To test solutions, they built a 6-component testbed (including a query router, Qdrant vector store, and LangGraph orchestrator). They intentionally injected 23 specific faults—such as silently swapping embedding models or corrupting JSON serialization—to measure how well different "resilience patterns" (like circuit breakers or quality gates) mitigated the damage.

Datasets

The study relies on a newly curated incident corpus of 150 reports:

  • Open-source: 97 incidents from 12 major projects (e.g., AutoGen, Haystack, DSPy).
  • Enterprise: 53 anonymized reports from production RAG and agent pipelines collected between 2024 and 2025.
  • Practitioner Resource: The authors have released the incident taxonomy as a structured JSON schema and provided the fault injection scripts to the community.

Key Insights and Ideas

  • The "Silent" Killer: 51% of incidents involved "silent degradation" where the system stayed up but produced wrong results. These take an average of 4.2 days to detect, compared to just 12 minutes for hard crashes.
  • The Error Multiplier: Retrieval failures have a 2.3x multiplier on downstream generation errors. When retrieval quality drops, generators don't just fail; they hallucinate more aggressively to try and make sense of the bad context.
  • Semantic Circuit Breakers: Traditional circuit breakers trip on 500 errors. AI circuit breakers must trip on semantic metrics (e.g., if retrieval relevance scores drop below a threshold), preventing a local failure from cascading into a system-wide "infinite retry" loop that drains API budgets.
  • Type Coercion is Risky: Even simple data handoffs (e.g., Python to TypeScript) caused "insidious" failures, such as floating-point precision shifts that broke strict equality checks and dropped 23% of results.

Key Results

The authors evaluated five resilience patterns with impressive quantitative results:

  • Circuit Breakers: Reduced the depth of error cascades by 89%.
  • Output Quality Gates: Caught 73% of silent degradation incidents before they reached the user.
  • Typed Interfaces (Pydantic): Eliminated 92% of integration failures by enforcing strict contracts at component boundaries.
  • Component Isolation: Reduced the "blast radius" of a failure (the % of affected concurrent requests) by 64%.
  • The "Power of Three": Systems implementing three or more of these patterns saw a 71% reduction in MTTR, bringing recovery time down from ~29 minutes to ~8 minutes.

Limitations and Future Work

  • Baseline Comparison: The 71% MTTR improvement was measured against "unstructured monitoring" (a relatively low bar). The authors admit that comparing against standard "retry-with-backoff" strategies is a necessary next step.
  • Scale: The experiments were conducted on a 6-component testbed. Real-world enterprise systems often involve 20+ components and nested agent hierarchies, which may introduce even more complex failure modes.
  • Overhead: These patterns aren't free. Quality gates add a median of 120ms of latency per request, and component isolation increases peak memory usage by about 18%. Future work is needed to optimize these "safety taxes" for latency-sensitive applications.

Summary updated

Abstract

Deploying compound AI systems reliably and safely requires understanding failure modes that emerge at component boundaries, not within individual models. Cascading errors propagate across component boundaries, silent quality degradation evades standard monitoring, and coordination failures yield incorrect collective behavior from individually correct parts. We analyze 150 production incident reports from open-source compound AI projects and anonymized enterprise deployments to construct a taxonomy of 23 failure modes organized into five categories: retrieval failures, generation failures, tool failures, orchestration failures, and integration failures. For each category, we propose resilience patterns with measured effectiveness from controlled fault injection experiments. Circuit breakers reduce cascade propagation by 89%, output quality gates catch 73% of silent degradation before user impact, and component isolation reduces blast radius by 64%. Systems implementing three or more resilience patterns from our catalog reduce mean-time-to-recovery (MTTR) by 71% compared to unstructured monitoring baselines. We release the incident taxonomy and pattern catalog as a practitioner resource.
BibTeX
@inproceedings{paul2026compound,
  author = {Rudrendu Kumar Paul and Sourav Nandy},
  title = {{Compound AI System Reliability: A Failure Taxonomy and Resilience Pattern Catalog from 150 Production Incidents}},
  booktitle = {AIWILD @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=06K05My3td},
}
Figure