TL;DR
RagHarnessDebugger is an open-source framework designed to audit and debug "agentic" Retrieval-Augmented Generation (RAG) systems. By using a fast, specialized classifier called Jev to guide and record every self-repair decision, it exposes "repair churn"—a phenomenon where system improvements are masked by new errors—enabling precise policy tuning for high-stakes tasks like legal contract analysis.
What Problem Does This Paper Address?
Agentic RAG systems are designed to be self-correcting: if an initial answer seems wrong or unsupported, the agent can fetch more evidence, revise its reasoning, and try again. However, evaluating these systems is notoriously difficult. Standard metrics like aggregate accuracy are "lossy"; they don't show if a system is actually getting better or if it’s just fixing one error while introducing another (a problem the authors call repair churn).
In the legal domain, where explainability and provenance are non-negotiable, we need to know why a system decided to change an answer or why it failed to find a repair that was clearly available in the text. This paper tackles the "black box" nature of RAG repair journeys by providing a verifiable audit trail for every micro-decision an agent makes.
Methods
The authors introduce a two-part system: a Jev-guided repair harness and an offline audit layer.
The Repair Harness
The harness (shown in Figure 1) moves through six stages:
- Retrieval: Uses BM25 and Jev to fetch and rerank evidence.
- Reading: Generates an initial answer ($A_0$) and performs "blind" rereads to check for consistency.
- Reading Repair: Conducts structured reading and diagnostic checks.
- Evidence Repair: If Jev detects missing info, the system fetches new passages ($E_1$).
- Validation: Checks if candidates are mechanically valid and supported by citations.
- Adjudication: A final LLM judge (or a code-based rule) selects the best answer, potentially applying a "veto" if internal checks fail.
Jev: The "System One" Guide
Instead of using expensive LLMs for every small decision, the authors use Jev, a fast, decision-only classifier. Jev acts like a "System One" (intuitive/fast) model that reranks evidence and flags when a retrieved package is likely insufficient, guiding the more expensive "System Two" (analytical/slow) LLM readers.
The Audit Layer
The debugger records every request, response, and state change in a hash-linked event chain. The offline audit layer (Figure 2) then verifies these traces without needing reference labels or further model calls, checking for "reachability" (could the system have even found the right answer?) and "intervention capability."
Datasets
The system was evaluated using ContractNLI, a dataset of non-disclosure agreements (NDAs). The task is Natural Language Inference (NLI): determining if a contract supports, contradicts, or does not mention a specific hypothesis.
- Development Set: 400 cases (800 executions) from 100 contracts.
- Replication Set: 374 cases (748 executions) from 22 separate contracts to test generalization.
Key Insights and Ideas
- Repair Churn is Real: In early versions of the harness, the system corrected many errors but introduced almost as many new ones. Aggregate accuracy looked flat, but the debugger revealed a high-activity "churn" underneath.
- The Acceptance Gap: The authors found that even when a correct answer was "reachable" (generated during the process), the system's own safety gates often blocked it. As shown in Figure 3, the harness missed roughly 76-77% of available repairs.
- Label-Free Triage: The debugger can identify "missed repairs" using label-free flags (like the "blocked conclusion" flag) with high precision. This allows developers to find where the system broke down without needing a human to grade every single output first.
- Veto Costs: While "vetoes" (blocking an answer because of a citation failure or reader disagreement) prevented 80-87% of potential new errors (regressions), they also blocked about 60% of potential repairs. The debugger allows researchers to replay these policies to find the optimal balance.
Key Results
By using the debugger to identify and fix flaws in the gating logic and repair-regression trade-offs, the authors produced a revised harness that significantly outperformed baselines:
| Condition | Dev Acc (%) | Repl Acc (%) |
|---|---|---|
| BM25 Baseline | 77.75 | 77.01 |
| Jev-guided ($A_0$) | 80.25 | 79.81 |
| Revised Harness | 81.38 | 80.75 |
| Whole Contract (Ref) | 80.25 | 78.34 |
- Accuracy Boost: The revised harness improved accuracy by 3.6–3.7 points over the BM25 baseline and even beat the "Whole Contract" reference score (which has access to all text but suffers from "lost in the middle" issues).
- Regression Reduction: The regression rate (turning a right answer into a wrong one) dropped from 5.3% to 0.9% on the development set after debugging.
- Cost Efficiency: The revised version reduced API costs by over 30% by optimizing when and how repairs were triggered.
Limitations and Future Work
The authors acknowledge that their findings are currently centered on the ContractNLI dataset and NDA-style logic. While the debugging framework is general, the specific failure patterns might differ in other domains. Additionally, they noted that "follow-up loops" (repeated attempts to fix a disputed reading) rarely changed the final label in this specific setup, suggesting that the initial candidate generation is often the primary bottleneck.
Future work includes deploying the auditor at runtime for live monitoring and testing the framework across a broader variety of legal tasks and model families.
Summary updated