VIGIL: A Reflective Runtime for Self-Healing LLM Agents

Christopher CruzAIWILD @ ICML 2026 · OpenReview

TL;DR

LLM agents often fail in subtle, cross-session ways that current monitoring tools miss. VIGIL introduces a novel "reflective runtime" that continuously observes agent behavior, diagnoses emergent issues, and proposes targeted fixes, acting as an X-ray machine for LLM fleets. It's the first infrastructure primitive for maintaining LLM agent health beyond individual sessions.

What Problem Does This Paper Address?

Deploying fleets of LLM agents presents a unique challenge: they can fail in ways that are invisible to traditional monitoring systems. These failures aren't usually single, catastrophic events but rather gradual "behavioral drift," creeping latency, or entirely new classes of errors that only emerge after an agent has been running for a while and interacting with its environment across multiple sessions. Existing tools either rely on pre-defined rules (AgentSpec), only monitor within a single session (Wink), or only flag major Service Level Agreement (SLA) violations (ARM). None of these can accumulate evidence over time or detect these novel, emergent failure modes. This gap leaves LLM agent fleets vulnerable to silent degradation and undetected issues.

Methods

VIGIL operates as an out-of-band "maintenance runtime," distinct from the LLM agent's primary task execution. Its core components include:

  • Deterministic Appraisal Engine: This engine processes structured event logs from the agent, converting each event into a diagnostic signal with attributes like label (e.g., "degraded," "recovered"), valence, intensity, and cause. This process is deterministic, meaning it's based on code, not stochastic LLM outputs.
  • Behavioral Signal Bank (EmoBank): A persistent log that stores these diagnostic signals. It uses exponential decay to prioritize recent events and employs coalescing policies to group similar events and rebound injection to detect recovery. This bank accumulates a historical view of the agent's behavior.
  • Roses/Buds/Thorns (RBT) Diagnosis: Based on the aggregated signals in EmoBank, VIGIL generates a diagnosis. Signals are categorized into "Roses" (positive), "Buds" (neutral/low-confidence positive), or "Thorns" (negative/degradation). This diagnosis is then mapped to domain-specific rules to identify systematic failures.
  • Stage-Gated Orchestration: VIGIL uses the LLM only for synthesizing proposed fixes (unified diffs) within a controlled runtime. These proposals are not applied automatically but are presented to a human operator for review, acting as a senior engineer suggesting a patch. The entire process is orchestrated through a strict stage machine to ensure predictable behavior.

Datasets

  • Synthetic Fleet Evaluation: A custom-built synthetic environment with 20 agents running for 7 episodes, featuring injected "structural failures" (consistent across episodes), "drift failures" (gradually worsening over episodes), and "novel failures" (appearing only later). This setup was designed to mimic real-world deployment patterns.
  • Nebius SWE-agent Dataset: A dataset of 100 real-world trajectories from the Nebius SWE-agent project, where agents attempted to resolve GitHub issues but failed to produce a valid patch. This dataset was used to evaluate VIGIL on uncontrolled, real-world agent failures across diverse codebases.

Key Insights and Ideas

  • The Maintenance Layer is Crucial: Just as software infrastructure evolved to include monitoring and diagnostics, LLM agent fleets now require a dedicated "maintenance runtime" to ensure their health and reliability.
  • Cross-Session Accumulation is Key: Subtle degradations and novel failures are often invisible within a single session. VIGIL's ability to accumulate behavioral evidence over extended periods is its primary advantage.
  • Deterministic Diagnostics, Stochastic Synthesis: VIGIL separates the reliable, architectural diagnosis of issues from the potentially stochastic generation of fixes. This ensures diagnostic reliability while leveraging LLMs for creative problem-solving.
  • LLM-as-Judge Bias is Real: The evaluation highlighted a significant inflation of correctness scores when using GPT-4o to judge its own outputs, underscoring the need for independent evaluation methods.
  • Behavioral Patterns, Not Just Bugs: VIGIL focuses on identifying patterns in agent behavior that indicate degradation, rather than pinpointing specific code bugs. This architectural choice is intentional and allows for broader applicability.

Key Results

  • Novel Failure Detection: In a comparative fleet evaluation, VIGIL was the only system to detect novel failure classes.
  • Earlier Drift Detection: VIGIL detected drift failures 3 episodes earlier than the ARM system.
  • Zero False Positives: Across all evaluated systems, VIGIL maintained a 0% false positive rate.
  • Real-World Validation: On 100 real-world SWE-agent trajectories, VIGIL-generated diffs were 100% syntactically valid and executable in a clean environment.
  • Richer Remediations: A longitudinal ablation study showed that accumulating more signals (142 rows vs. 23 rows) led to measurably richer and more targeted remediation proposals.
  • GPT-4o Self-Inflation: A cross-model validation study revealed GPT-4o self-inflation of +1.05 points compared to Claude.

Limitations and Future Work

  • Specificity Gap: VIGIL's proposals address behavioral patterns, not specific code bugs within a repository. Closing this gap by integrating VIGIL with a downstream coding agent is identified as future work.
  • Synthetic Evaluation: While derived from real patterns, the initial fleet evaluation used a synthetic environment. The real-trajectory evaluation in Setting 2 helps mitigate this.
  • Benchmark for Fleet Maintenance: The paper notes that a comprehensive benchmark for evaluating fleet-level maintenance tooling does not yet exist, and the presented settings are initial steps.
  • LLM for Synthesis: While VIGIL controls the LLM's role to diff synthesis, the quality of these synthesized diffs is still subject to LLM capabilities. Future work could explore more sophisticated LLM integration for fix generation.

Summary updated

Abstract

LLM agent fleets fail in ways that per-session monitors cannot detect: behavioral drift, latency creep, and novel failure classes that emerge only across sessions. No infrastructure primitive exists for cross-session fleet health monitoring. Existing approaches either require pre-declared failure rules (AgentSpec), operate within single sessions (Wink), or trigger only on coarse SLA violations (ARM)---none accumulate behavioral evidence across sessions or detect novel failure classes. We present \textbf{V.I.G.I.L} (\textit{Verifiable Inspection and Guarded Iterative Learning}), a reflective runtime that occupies this maintenance layer. VIGIL is not a coding agent. It does not resolve issues, compete on task benchmarks, or produce repo-specific patches. VIGIL ingests structured behavioral logs, appraises each event via a deterministic appraisal engine, maintains a persistent \textit{Behavioral Signal Bank (EmoBank)} with exponential decay and coalescing policies, and derives a \textit{Roses/Buds/Thorns (RBT)} diagnosis mapping fleet behavior into strengths, opportunities, and systematic failures. Appraisal, decay, coalescing, threshold routing, and stage enforcement are all deterministic code. The LLM is used only for diff synthesis inside a governed runtime. Diagnostic reliability is architectural, not stochastic. We make two primary contributions. First, we identify and formalize the fleet-level agent maintenance problem, including a taxonomy of failure classes and the information-boundary constraints that determine what each class of system can detect. Second, we propose and evaluate VIGIL as a concrete instantiation of this primitive across three complementary settings. In a comparative fleet evaluation (20 agents $\times$ 7 episodes, 3 seeds), VIGIL is the only system to detect novel failure classes (episode 4); ARM misses drift for 5 episodes; AgentSpec and ARM achieve 0\% novel detection. False positive rate is 0\% across all four systems. Detection behavior is identical across all three seeds, confirming that VIGIL's diagnostic properties are architectural rather than stochastic. In a real-trajectory evaluation on 100 Nebius SWE-agent instances, Claude-as-judge correctness scores 4.20/5. Execution-based verification confirms 100/100 (100\%) of VIGIL-generated diffs are syntactically valid Python and execute successfully in a clean environment across all 100 distinct repositories. A cross-model validation study reveals GPT-4o self-inflation of $+1.05$ points ($t=34.0$, $p < 0.0001$). A longitudinal ablation confirms 142 accumulated signal rows produce measurably richer remediations than 23 rows from the same instance.
BibTeX
@inproceedings{cruz2026vigil,
  author = {Christopher Cruz},
  title = {{VIGIL: A Reflective Runtime for Self-Healing LLM Agents}},
  booktitle = {AIWILD @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=bE8nRXcWC1},
}

Loading related content...

Figure