Telling Distraction Apart from Ignorance in Long-Running Domain Agents

When a language-model agent works through a long technical task, such as tuning a training run over many rounds or solving a chain of physics sub-problems, it can fail at a given step for two very different reasons. Either it never knew what to do (a real knowledge gap), or it did know but was pulled off course by the clutter it has accumulated: old logs, earlier guesses, retrieved documents. Current methods treat both as one thing, "the model is uncertain here." The usual response is to retrieve more documents or to abstain. That is the right fix for ignorance and the wrong fix for distraction, because retrieval adds still more text to a context that is already the source of the problem.

We propose a cheap, per-step test that separates the two cases. At each step we ask the agent the same question twice: once with its full history, and once with a stripped-down view that keeps only the task, the best result so far, and the latest observation. If the stripped-down agent is confident and the full-history agent disagrees, the history is distracting it, so we act on the clean view. If the stripped-down agent is itself unsure, the gap is real, so we retrieve targeted domain knowledge or flag the step as beyond the agent's competence. The claim is that matching the remedy to the cause beats applying any single remedy everywhere, and that this matters most in long-horizon, specialized-domain work, where histories grow and gaps are common.

Core Claim

Work on "knowledge boundaries" (whether a model can tell what it knows from what it doesn't) estimates that boundary as a property of the question alone. Examples include probing known versus unknown splits in BARREL, kNN over question embeddings in Self-Knowledge Guided Retrieval, and step-level over-search and under-search detection in HiPRAG. The distraction literature shows the boundary actually moves with context. Breaking Focus finds that models lose about 45% accuracy under coherent distractors on questions they answer correctly when clean; GPT-4o-mini on CommonsenseQA falls from 85.7% to 22.0%. Thinking in a Crowd shows that step-by-step reasoning rationalizes misleading snippets rather than filtering them. In a long-horizon loop, the agent's own history is the distractor. AgentHPOBench reports that agents find a good configuration and then abandon it after over-reacting to noisy intermediate logs.

We therefore treat the boundary as a function of both the step's question and its context, and we read it off the gap between two views of the same step. Let $h_t$ be the full history at step $t$ and $c(h_t)$ a fixed compressed view. Draw $K$ samples of the next action from each view and canonicalize them, for example as a hyperparameter configuration or a numeric answer. Let $q_c$ be the frequency of the most common action under $c(h_t)$, and $A$ the fraction of full-history samples that agree with it. The diagnostic places each step in one of three cells. High $q_c$ with high $A$ means act. High $q_c$ with low $A$ means distraction, so act on the compressed view. Low $q_c$ means a knowledge gap, so issue a retrieval query built from the step and re-probe on the compressed view plus the retrieved passages. If $q_c$ is still low after retrieval, the agent emits a localized "unsure at this step" flag in the spirit of HALT and continues on independent branches. Thresholds are set on a development split.

This proposal does not build a better noise filter or a better abstention model. It argues that a step-level, context-conditioned diagnosis is what makes existing filters and abstention methods usable inside long agent loops. Its central falsifiable prediction is that retrieval at steps labeled "distraction" should hurt.

Research Program

The work has three parts.

The first is a controlled benchmark where the cause of each failure is known by construction. We chain problems from TheoremQA and SciAux into multi-step trajectories in which later steps reuse earlier results. For each step we probe the clean question many times, following BARREL's multi-trial procedure, to label it known or unknown for the model under test. We then grow the history with three kinds of filler: neutral prior steps, SciAux irrelevant and misleading snippets, and distractors found by the error-guided search of Breaking Focus. A known step that fails under history is ground-truth distraction. An unknown step that fails clean is a ground-truth gap. This lets us score the diagnostic directly, separately from its downstream effect.

The second part is the agent setting. We run the diagnostic inside AgentHPOBench's 30 executable machine-learning repositories. Here the history is the agent's own training logs, the domain gap is practical ML-engineering knowledge, and the failure of interest (abandoning a good configuration) is observable. We extend the five-turn horizon to ten turns to stress history growth, and also report the native five-turn protocol for comparability.

The third part amortizes the cost. The probe needs $2K$ samples per step. We train a small classifier on the agent's hidden states at the step boundary to predict the diagnostic cell, using probe outcomes as labels, and test whether it keeps most of the gain at a single forward pass.

The four input directions play different roles. Distraction is the mechanism we diagnose, knowledge boundaries are the quantity we redefine, long horizons are the regime where both collide, and specialized domains supply real gaps so the "unknown" cell is populated.

Baselines

The do-nothing reference is the same agent with full history and no intervention. On AgentHPOBench, the paper reports a Baseline Win Rate of 76.7% and Mean Anchor Attainment of 79.5% for Claude Sonnet 4.6. It describes Qwen3-32B as the strongest open-weight agent but gives no number in the file. We will rerun the open-weight models ourselves under the identical harness. On the controlled benchmark, the closest reported reference is SciAux's Qwen3-8B, which scores 96.22% clean and 77.03% with two misleading snippets with reasoning on.

The first naive competitor is always-compress: every step uses only $c(h_t)$. This is a serious threat. If history is mostly harmful, it should capture most of our gain for free.

The second is a best-so-far memory baseline for AgentHPOBench: the full history plus an explicit pinned record of the best configuration and score. It targets the non-monotonic failure directly.

The third is entropy-gated retrieval. It retrieves whenever full-history sampling disagreement exceeds a threshold, which is the question-only boundary view in prompt form. It also stands in for When Do LLMs Need Retrieval Augmentation?, whose Punish+Explain prompting reached 0.676 accuracy on Natural Questions with 33.2% retrieval calls using GPT-4.

The strongest trained competitor is HiPRAG. It uses step-level process rewards that penalize unnecessary and missing searches, and reports 67.2% average multi-hop QA accuracy and a 2.3% over-search rate at 7B. No reported result was found on Lacuna for HiPRAG on AgentHPOBench, TheoremQA, or SciAux. We will train it on our trajectories with the same base model, the same retriever and corpus, and matched GPU hours. A MASH-style pay-per-search reward from Pay-Per-Search Models are Abstention Models is a secondary trained baseline; it reports 59.9% no-tool precision on HotPotQA.

All inference baselines get the same total sample budget per step as our probe. For example, entropy gating uses $2K$ full-history samples, and always-compress uses $2K$ samples with majority vote. Thresholds for every method are tuned on the same development split.

Two ablations isolate the key ingredient. The first, a remedy swap, retrieves in distraction cells and compresses in gap cells. The second, gap-only, keeps retrieval in gap cells but never switches to the compressed view.

Beating the field means three things. We must exceed the best of always-compress and best-so-far memory on Mean Anchor Attainment with non-overlapping bootstrap intervals across the 30 tasks. We must match or exceed HiPRAG at lower training cost. And we must show the remedy-swap ablation doing worse than either correct remedy.

Why not just…?

Why not just always compress the history? It may work, and we treat that as a live outcome. The file gives reasons to expect it to fall short. Topological isolation is known to sever genuine cross-step dependencies (the inter-document interference problem noted for Superposition Prompting). Our chained TheoremQA steps deliberately require earlier results that a fixed compressor may drop. The decisive comparison is the high-$A$ cell: if acting on compressed views there performs no worse than on full histories, the diagnostic adds nothing and we will report that the problem reduces to context hygiene.

Why not just gate retrieval on the model's uncertainty, as SKR or Punish+Explain do? Uncertainty under the full history cannot tell whether the confusion comes from the context or from missing knowledge. In the distraction case, retrieval appends text to an already harmful context. UDA shows how badly clutter hurts domain numerics (GPT-4 long-context 0.4 versus 45.9 with targeted retrieval on FinHybrid). The controlled benchmark measures this directly: entropy gating should fire retrieval on ground-truth distraction steps, and those steps should not improve.

Why not just train with step-level search rewards like HiPRAG or MASH? These methods learn when search is needed against a question-level notion of knowing. They were trained on clean open-domain QA histories. They might nonetheless learn implicitly to discount history, which would make them a strong competitor. Our matched-compute comparison on the same trajectories settles this. We also check whether HiPRAG's over-search detections cluster on our distraction cells, which would mean it mislabels distraction as ignorance.

Why not just use a bigger model with a longer context? Distraction persists at the frontier. Breaking Focus finds it in leading models, and AgentHPOBench's best agent still regresses across turns. We include one frontier API agent with full history as a reference point. We expect its regression rate to remain well above zero.

Proposed Experiments

On the controlled benchmark, the primary metric is diagnostic accuracy: macro-F1 over the three cells against constructed labels, compared with entropy-only labeling. Secondary metrics are end-task accuracy and reasoning-token count. BARREL's finding that wrong answers on QwQ-32B are 136.5% longer suggests that correct routing should also shorten traces.

On AgentHPOBench, we report Baseline Win Rate, Mean Anchor Attainment, and a regression rate: the fraction of runs whose final configuration is worse than one the agent had already found. We report each at five and ten turns, expecting the gap over the do-nothing reference to widen with horizon.

Three results decide the proposal. The first is the remedy-swap ablation: retrieval in distraction cells must measurably lower accuracy relative to compressing. If it does not, the two-cause premise is wrong. The second is the always-compress comparison, which tells us whether diagnosis earns its sampling cost. The third is whether the hidden-state classifier recovers most of the probe's gain at one forward pass, which determines whether the method is practical inside long loops.