RagHarnessDebugger: Jev-Guided Agentic RAG in Contract Inference Repair Auditing

Moriya Dechtiar, Daniel Martin Katz, Michael J. Bommarito, Hongming Wang, Sylvain JaumearXiv 2026 · arXiv

TL;DR

RagHarnessDebugger is an open-source framework designed to audit and debug "agentic" Retrieval-Augmented Generation (RAG) systems. By using a fast, specialized classifier called Jev to guide and record every self-repair decision, it exposes "repair churn"—a phenomenon where system improvements are masked by new errors—enabling precise policy tuning for high-stakes tasks like legal contract analysis.

What Problem Does This Paper Address?

Agentic RAG systems are designed to be self-correcting: if an initial answer seems wrong or unsupported, the agent can fetch more evidence, revise its reasoning, and try again. However, evaluating these systems is notoriously difficult. Standard metrics like aggregate accuracy are "lossy"; they don't show if a system is actually getting better or if it’s just fixing one error while introducing another (a problem the authors call repair churn).

In the legal domain, where explainability and provenance are non-negotiable, we need to know why a system decided to change an answer or why it failed to find a repair that was clearly available in the text. This paper tackles the "black box" nature of RAG repair journeys by providing a verifiable audit trail for every micro-decision an agent makes.

Methods

The authors introduce a two-part system: a Jev-guided repair harness and an offline audit layer.

The Repair Harness

The harness (shown in Figure 1) moves through six stages:

  1. Retrieval: Uses BM25 and Jev to fetch and rerank evidence.
  2. Reading: Generates an initial answer ($A_0$) and performs "blind" rereads to check for consistency.
  3. Reading Repair: Conducts structured reading and diagnostic checks.
  4. Evidence Repair: If Jev detects missing info, the system fetches new passages ($E_1$).
  5. Validation: Checks if candidates are mechanically valid and supported by citations.
  6. Adjudication: A final LLM judge (or a code-based rule) selects the best answer, potentially applying a "veto" if internal checks fail.
Fig. 1. The repair harness. Solid arrows are the open path; a closed gate returns the current answer or skips evidence repair (dashed gold), and dashed slate arrows return to adjudication after a follow-up branch. A deferral runs at most one follow-up branch, after which adjudication runs once more; a second deferral keeps A0 and is reported as unresolved. Every decision is recorded for the offline audit (Fig. 2).

Jev: The "System One" Guide

Instead of using expensive LLMs for every small decision, the authors use Jev, a fast, decision-only classifier. Jev acts like a "System One" (intuitive/fast) model that reranks evidence and flags when a retrieved package is likely insufficient, guiding the more expensive "System Two" (analytical/slow) LLM readers.

The Audit Layer

The debugger records every request, response, and state change in a hash-linked event chain. The offline audit layer (Figure 2) then verifies these traces without needing reference labels or further model calls, checking for "reachability" (could the system have even found the right answer?) and "intervention capability."

Fig. 2. The audit layer. Everything above the dashed line runs offline, without reference labels or model calls. Labels join only after the findings are frozen.

Datasets

The system was evaluated using ContractNLI, a dataset of non-disclosure agreements (NDAs). The task is Natural Language Inference (NLI): determining if a contract supports, contradicts, or does not mention a specific hypothesis.

  • Development Set: 400 cases (800 executions) from 100 contracts.
  • Replication Set: 374 cases (748 executions) from 22 separate contracts to test generalization.

Key Insights and Ideas

  • Repair Churn is Real: In early versions of the harness, the system corrected many errors but introduced almost as many new ones. Aggregate accuracy looked flat, but the debugger revealed a high-activity "churn" underneath.
  • The Acceptance Gap: The authors found that even when a correct answer was "reachable" (generated during the process), the system's own safety gates often blocked it. As shown in Figure 3, the harness missed roughly 76-77% of available repairs.
  • Label-Free Triage: The debugger can identify "missed repairs" using label-free flags (like the "blocked conclusion" flag) with high precision. This allows developers to find where the system broke down without needing a human to grade every single output first.
  • Veto Costs: While "vetoes" (blocking an answer because of a citation failure or reader disagreement) prevented 80-87% of potential new errors (regressions), they also blocked about 60% of potential repairs. The debugger allows researchers to replay these policies to find the optimal balance.
Breakdown of initial errors and their final outcomes, highlighting the large percentage of missed repairs.

Key Results

By using the debugger to identify and fix flaws in the gating logic and repair-regression trade-offs, the authors produced a revised harness that significantly outperformed baselines:

Condition Dev Acc (%) Repl Acc (%)
BM25 Baseline 77.75 77.01
Jev-guided ($A_0$) 80.25 79.81
Revised Harness 81.38 80.75
Whole Contract (Ref) 80.25 78.34
  • Accuracy Boost: The revised harness improved accuracy by 3.6–3.7 points over the BM25 baseline and even beat the "Whole Contract" reference score (which has access to all text but suffers from "lost in the middle" issues).
  • Regression Reduction: The regression rate (turning a right answer into a wrong one) dropped from 5.3% to 0.9% on the development set after debugging.
  • Cost Efficiency: The revised version reduced API costs by over 30% by optimizing when and how repairs were triggered.

Limitations and Future Work

The authors acknowledge that their findings are currently centered on the ContractNLI dataset and NDA-style logic. While the debugging framework is general, the specific failure patterns might differ in other domains. Additionally, they noted that "follow-up loops" (repeated attempts to fix a disputed reading) rarely changed the final label in this specific setup, suggesting that the initial candidate generation is often the primary bottleneck.

Future work includes deploying the auditor at runtime for live monitoring and testing the framework across a broader variety of legal tasks and model families.

Summary updated

Abstract

Agentic systems using retrieval-augmented generation (RAG) must repair their own answers by gathering more evidence, testing alternative policies and deciding if, when, and what repairs are needed. Yet how do we know if a system's repairs are working when metrics like accuracy can be misleading? We present RagHarnessDebugger, a framework that records every repair decision in a Jev-guided harness for contract inference and audits the record to facilitate complete debugging of all decisions taken during a RAG repair journey. RagHarnessDebugger checks that each trace is intact, that each validation applies to the candidate it is counted for, and which answers the acceptance rule can reach. We tested our approach on ContractNLI non-disclosure agreements with two reader models and a replication set of separate contracts. Our debugger's audit layer exposed failure patterns that were hidden under aggregate metrics, policies acceptance cost of repairs vs regressions and policy acceptance gap of 77% and 76% of available but unrealised repairs on development and replication sets. Our audit, together with a labeled error analysis and replay mechanism, allowed following the execution path all the way to the terminal decision - uncovering where missing repairs became lost. Our revised harness improved accuracy over the initial BM25-only baseline by 3.6 points on development contracts and 3.7 points on the replication set, and even improved our full contract access reference score by 1.1 on development and 2.4 on replication set. While our work here demonstrates how the debugger can conduct decision root-cause analysis for missed repairs, the true value lies with providing this debugger for judgment visibility serving continuous improvement of RAG decisions and their explainability. Keywords: agentic RAG, AI evidence repair, harness explainability, counterfactual replay, Jev, ContractNLI
BibTeX
@misc{dechtiar2026ragharnessdebugger,
  author = {Moriya Dechtiar and Daniel Martin Katz and Michael J. Bommarito and Hongming Wang and Sylvain Jaume},
  title = {{RagHarnessDebugger: Jev-Guided Agentic RAG in Contract Inference Repair Auditing}},
  year = {2026},
  eprint = {2611.22991},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22991},
}
Figure