Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention

Rakshith Vasudev, Melisa Russak, Dan Bikel, Waseem AlShikharXiv 2026 · arXiv · OpenReview

TL;DR

This paper reveals a critical tradeoff in LLM agent interventions: while accurate failure prediction is useful, it doesn't guarantee improved performance. Interventions can disrupt successful trajectories, leading to significant performance degradation, especially in high-success scenarios. The authors propose a practical pilot-based framework to assess this disruption-recovery tradeoff before deployment, preventing costly regressions.

What Problem Does This Paper Address?

Large Language Model (LLM) agents are increasingly used for complex, multi-step tasks. A common strategy to improve their reliability is to use a "critic" LLM to predict failures mid-execution and intervene. The assumption is that accurate failure prediction directly translates to better overall performance. However, the authors observe that this isn't always true; sometimes, these interventions actually harm performance. This paper investigates why this happens and proposes a method to predict whether an intervention will be beneficial or detrimental before deploying it in a real-world scenario.

Methods

The core of the paper's contribution lies in formalizing the "disruption-recovery tradeoff." They define:

  • Recovery Rate (r): The proportion of initially failing trajectories that are successfully recovered by intervention.
  • Disruption Rate (d): The proportion of initially successful trajectories that are disrupted (made to fail) by intervention.

They derive a simple condition: intervention is beneficial only when the baseline failure rate ($p$) is greater than a threshold derived from $r$ and $d$ ($p > d/(r+d)$). To estimate this tradeoff without full deployment, they propose a pre-deployment pilot study using a small set of tasks (around 50) to measure $p$, $r$, and $d$ for a specific agent-benchmark pair. They also explore two simple intervention mechanisms: ROLLBACK (undoing the last action and retrying) and APPEND (adding a warning message).

Datasets

The experiments were conducted on three benchmarks:

  • HotPotQA: A multi-hop question-answering dataset with relatively high baseline success rates.
  • GAIA: A benchmark for general AI assistants requiring reasoning and multimodal understanding, with medium success rates.
  • ALFWorld: A deterministic household robotics simulation with a high-failure rate.

The LLM critic models were trained on data from smolagents runs on HotPotQA and GAIA.

Key Insights and Ideas

  • Accuracy isn't everything: A highly accurate LLM critic (AUROC 0.94) can still lead to severe performance degradation. The critic's ability to predict failure is less important than how the agent reacts to that prediction.
  • The Disruption-Recovery Tradeoff is King: The core problem is that interventions can disrupt successful trajectories. The ratio of disruption to recovery ($d/r$) is a much stronger predictor of intervention success than the critic's raw accuracy.
  • High-success tasks are risky for intervention: In scenarios where agents are already performing well, interventions are more likely to cause harm by disrupting successful paths.
  • Low-success tasks offer limited upside: Even in high-failure scenarios, the gains from intervention are modest, suggesting a ceiling on how much prediction accuracy can help.
  • Early interventions are particularly damaging: Most regressions occur when interventions are applied very early in a trajectory (steps 0-1), often derailing already correct paths.
  • Model sensitivity varies wildly: The same intervention can catastrophically harm one agent while only mildly affecting another, highlighting the agent's internal dynamics as the bottleneck.

Key Results

  • Severe Regressions: In high-success regimes (HotPotQA, GAIA), interventions degraded performance by up to 26 percentage points on some models (MiniMax-M2.1).
  • Modest Gains: In a high-failure regime (ALFWorld), interventions yielded a modest improvement of +2.8 pp (p=0.014).
  • Pilot Study Effectiveness: The proposed pilot study correctly predicted whether intervention would help or harm across different benchmarks and models.
  • Oracle Limits: Even with perfect failure prediction (oracle intervention), the gains were limited (3-8 pp), significantly less than oracle post-hoc selection (11-17 pp), underscoring the "disruption tax."
  • Scaling Doesn't Solve It: A larger 14B critic model did not outperform a smaller 0.6B model, suggesting data diversity and agent response are the primary limitations, not critic capacity.

Limitations and Future Work

  • Limited Intervention Mechanisms: The study used simple rollback and append mechanisms. More sophisticated methods might mitigate disruption.
  • Benchmark and Agent Coverage: Experiments were on three benchmarks and three agent backbones. The findings might not generalize universally.
  • Statistical Power for Positive Results: The positive gain on ALFWorld was close to the detection limit, suggesting it's directional evidence rather than a precise estimate.
  • Pilot Transferability: The reliability of transferring pilot estimates from one domain (e.g., QA) to another (e.g., coding) is untested.
  • Future Work: The authors suggest exploring more targeted intervention methods, testing larger and more diverse LLM critics, and investigating how to improve agent robustness to mid-trajectory corrections. They also highlight the need for larger-scale validation of LLM critic-based selection.

Summary updated

Abstract

Proactive interventions by LLM critic models are often assumed to improve reliability, yet their effects at deployment time are poorly understood. We show that a binary LLM critic with strong offline accuracy (AUROC 0.94) can nevertheless cause severe performance degradation, inducing a 26 percentage point (pp) collapse on one model while affecting another by near zero pp. This variability demonstrates that LLM critic accuracy alone is insufficient to determine whether intervention is safe. We identify a disruption-recovery tradeoff: interventions may recover failing trajectories but also disrupt trajectories that would have succeeded. Based on this insight, we propose a pre-deployment test that uses a small pilot of 50 tasks to estimate whether intervention is likely to help or harm, without requiring full deployment. Across benchmarks, the test correctly anticipates outcomes: intervention degrades performance on high-success tasks (0 to -26 pp), while yielding a modest improvement on the high-failure ALFWorld benchmark (+2.8 pp, p=0.014). The primary value of our framework is therefore identifying when not to intervene, preventing severe regressions before deployment.
BibTeX
@misc{vasudev2026accurate,
  author = {Rakshith Vasudev and Melisa Russak and Dan Bikel and Waseem AlShikh},
  title = {{Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention}},
  year = {2026},
  eprint = {2602.03338},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.03338},
}

Loading related content...

Figure