Language-model agents that split a hard task into subtasks, and hand each subtask to a fresh, isolated model call, are much more robust to irrelevant text than agents that read everything in one long prompt. They are also much more expensive. A recursive planner on a kitchen-simulation benchmark costs roughly three times as many model calls as a flat agent (ReCAP). Current systems decompose either always or by fixed rule. This proposal argues that most of the accuracy benefit of decomposition comes from noise isolation: a subtask run in its own clean workspace never sees the distractors that would have pulled the model's attention off course. If that is true, decomposition should be paid for only at the steps where the context would actually distract the model, and those steps can be detected cheaply from signals the model produces while it reads the context anyway.
Concretely, we would build a controller that sits inside a planner–executor agent. At each step it runs the single prefill pass that a flat answer needs. A prefill pass is the forward pass that reads the prompt before any tokens are generated. From that pass the controller computes a small set of distraction features, chiefly how much of the query's attention lands on passages that a relevance scorer judges irrelevant. It then decides whether to answer flat, prune and answer, or spawn an isolated sub-executor. The target is to recover most of the accuracy of always-decompose agents at a fraction of their calls, tokens, and memory. The project also produces a measured curve of "decomposition gain versus context noise", which no paper in this area reports.
Core Claim
Our hypothesis is that the gain from isolating a subtask in its own workspace is predicted by the noise level of the context that subtask would otherwise share, and much less by context length or query complexity. We further hypothesize that a cheap attention-and-relevance signal taken from the flat prefill predicts this gain well enough to support per-step decisions.
The controller decomposes step $t$ when $\hat g_t > \lambda c_t$. Here $\hat g_t$ is the predicted accuracy gain from isolating the step, $c_t$ is its marginal cost in prefill-plus-decode tokens (or measured GPU-seconds), and $\lambda$ is a user-set price that traces out the accuracy–cost frontier.
The claim is new because each neighbouring line of work fixes one of these choices. CoDA always isolates raw documents in an executor workspace and never measures cost (CoDA). ReCAP always decomposes, and its authors report the threefold cost themselves. SymRAG routes on query complexity and system load, but never on the content of the retrieved context (SymRAG). Superposition Prompting prunes non-salient documents in a single flat step and has no notion of subtasks (Superposition Prompting).
Why the signal should work, and a tension it turns into a feature
Several papers motivate the hypothesis. Distractors hurt models by attracting attention and confident wrong answers rather than by sheer length. Zero-shot readers degrade as more documents are retrieved, and they stay overconfident on irrelevant passages (Improving Zero-shot Reader). Code models latch onto narrative nouns in long problem statements (Less is More). Long-context GPT-4 collapses from 45.9 to 0.4 on the UDA FinHybrid financial-document task when it is given the whole report instead of retrieved chunks (UDA).
DefensiveKV is a cache-eviction method that keeps any token which ever received a spike of attention (Taming the Fragility of KV Cache Eviction). Taken together with the distractor findings, its rule implies something uncomfortable: distractors are exactly the tokens that win attention spikes, so defensive retention would protect them. We use that same property as a detector. When the attention mass from the query's final tokens concentrates on spans with a low query-likelihood relevance score, we predict the flat answer will be hijacked. The relevance score is $P(\text{query}\mid\text{span})$, the query-generation score from DAS, a distraction-aware answer-selection method. We also include redundancy among the attended spans, in the spirit of R-KV, and attention entropy over spans.
Research Program
The work has three parts.
The first is a measurement study. On multi-hop question answering, we take each question's gold supporting passages and pad them with retrieved distractors. We do this so that total context length and distractor fraction vary independently on a grid. For every cell we run the same model flat and with an isolated executor. This gives a direct estimate of decomposition gain as a function of noise at fixed length, and of length at fixed noise. If gain tracks length rather than noise, the core claim is wrong, and we would report that.
The second part is the controller. Its features come from the flat prefill, plus relevance scores computed with the same model's log-likelihoods. We label a held-out calibration split by running both branches, and fit a small logistic or gradient-boosted regressor that predicts $\hat g_t$. The cost $c_t$ is estimated from token counts and checked against measured latency. The controller has three actions: answer flat, prune low-relevance spans and answer flat (a Superposition-style prune), or spawn an executor. When the controller decomposes, it reuses the cache from the prefill it already ran wherever possible, so the probe is not a wasted pass.
The third part is the agent. We embed the controller in a planner–executor agent modelled on CoDA's structure. Because CoDA's reinforcement-learning training (PECO) may not be reproducible from the file alone, the main comparisons use a prompted planner–executor. CoDA reports that its trained version beats this kind of prompted hierarchy by 8 F1 points. If CoDA weights are available, we also run the controller on top of them.
Baselines
The naive reference is a flat retrieval-augmented reader that concatenates all retrieved passages into one prompt. Everything else must beat it on accuracy, and it is the cheapest point on the cost axis.
The strongest flat competitor is DAS, which combines an instruction to answer "unanswerable" when unsure with query-generation-score reweighting. Its paper reports an average 64% exact-match increase over naive zero-shot readers (Improving Zero-shot Reader). We include Superposition-style path pruning, which reports 43% accuracy gains and 93-fold compute reduction over naive RAG on Natural Questions with MPT-7B. We reimplement only its pruning logic on our backbone so that the comparison is not confounded by its position-embedding scheme.
The strongest decomposition competitor is always-decompose. This is a CoDA-style planner–executor that sends every retrieval step to an isolated executor. CoDA reports relative F1 gains of +10.65% on MuSiQue and +24.39% on 2WikiMultiHopQA over AutoRefine, and stability where AutoRefine loses 52% F1 as context grows. We found no absolute F1 values on Lacuna for these CoDA results, so we will report our own reproduced numbers. For the agent setting, the reference is ReCAP: 70% synchronous and 53% asynchronous success on Robotouille at about three times ReAct's cost.
Fairness rules apply to every system. All systems use the same backbone, retriever, top-k, and decoding settings. All run on the same serving stack with prefix caching enabled, so that no system wins by re-prefilling shared prompts less. The controller's calibration split is disjoint from test, and baseline prompts receive the same tuning budget, measured in prompt variants tried on the same split.
Beating the competitors means one of two outcomes. Either the controller matches always-decompose accuracy within one F1 point at no more than half its model calls and tokens, or it strictly dominates DAS and pruning on the accuracy–cost frontier at every price $\lambda$ tested.
Two ablations isolate the key ingredient. The first replaces the noise features with context length alone. The second replaces them with SymRAG-style query-only complexity features. Both keep the same controller and decision rule. If either matches the full controller, the noise signal is not what matters.
Why not just…?
Why not just always decompose, as CoDA and ReCAP do? Always-decompose is the accuracy ceiling we aim to approach, not beat. Its weakness is cost: ReCAP reports roughly threefold expense, and CoDA reports none. The measurement study shows whether many steps gain nothing from isolation. If gains are uniform across noise levels, the controller has nothing to save, and the honest conclusion would be that decomposition should simply be used everywhere.
Why not just decompose when the context is long? This is the real threat, because noise and length are correlated in natural retrieval. The fixed-length, variable-noise grid is designed to separate them, and the length-only ablation runs the identical controller. If the length rule matches the full controller on natural benchmarks, we would recommend the length rule and treat the noise signal as unnecessary in practice.
Why not just filter the context and answer flat, with DAS or pruning? Flat filtering is cheap and strong on single-hop questions. The evidence suggests it caps out on multi-hop dependencies. Summarization and condensing help easy programming problems but yield only about 2% on competition-level ones (Less is More). Superposition Prompting is restricted by design to flat, single-step retrieval. Our controller includes prune-and-answer as an action, so if filtering suffices, the controller should learn to choose it, and the frontier will show this.
Why not just route by query difficulty, as SymRAG does? SymRAG's router never looks at the retrieved text. The same question can be easy with clean passages and fail with distractor-heavy ones. The query-only ablation tests whether that context-blindness matters.
Proposed Experiments
The main benchmarks are MuSiQue, 2WikiMultiHopQA, and HotpotQA under the controlled distractor grid and under natural retrieval. We add UDA FinHybrid as a test of numerically dense documents. Its reported RAG score of 45.9 against 0.4 for full long context marks the two extremes the controller must navigate. Robotouille serves as a transfer test to an interactive agent, using a ReCAP-style tree with our controller gating child spawns.
Metrics are F1 and exact match, or success rate for Robotouille. Cost is measured as model calls, prefill and decode tokens, end-to-end latency, peak KV-cache memory (the memory holding each token's stored attention keys and values), and time to first token. Results are reported as accuracy–cost frontiers swept over $\lambda$.
Three checks are decisive. First, on the controlled grid, decomposition gain must rise with distractor fraction at fixed length; if it does not, the premise fails. Second, the controller's predicted $\hat g_t$ must rank steps by realized gain with clearly better calibration than the length-only and query-only predictors on held-out data. Third, on natural benchmarks, the controller's frontier must reach within one F1 point of always-decompose at half or less of its calls, and must not lose to DAS at any cost level. A secondary analysis checks whether the attention-on-irrelevant-spans feature transfers across backbones without recalibration. Transfer would make the method usable on top of trained hierarchies like CoDA, not only prompted ones.
References
- ReCAP: Recursive Context-Aware Reasoning and Planning for Large Language Model Agents
- CoDA: A Context-Decoupled Hierarchical Agent with Reinforcement Learning
- SymRAG: Efficient Neuro-Symbolic Retrieval Through Adaptive Query Routing
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Less is More: Summary of Long Instructions is Better for Program Synthesis
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- DefensiveKV: Taming the Fragility of KV Cache Eviction in LLM Inference
- R-KV: Redundancy-aware KV Cache Compression for Reasoning Models