A proactive assistant speaks before it is asked. It might surface a document in a conversation, a file to a data analyst, or a fix to a programmer. Current work treats the question of when to speak as a property of the user's situation: is the user stuck, idle, or receptive? But when such an assistant draws on a large document store, many of its worst interruptions have a different cause. It picked the right moment and then handed over the wrong document, a near-miss that looks relevant and is not. A near-miss distractor confidently delivered at a good moment is worse than silence, because it costs the user's attention and also misleads them. We propose a gate that decides whether to interrupt by looking at the user's context and the evidence it is about to deliver. We train it on deliberately confusing counterexamples: the right moment paired with a near-miss document, and the right document paired with a moment that is slightly too early.
The design follows a slogan that settles a tension in the literature. Retrieval that runs silently in the background should be cheap and frequent, triggered by the system's own uncertainty, as in active retrieval for generation. Delivery to the user should be rare, and should happen only when a separate judge is confident that this specific piece of evidence, at this moment, will be accepted. The retriever and the judge are kept as separate components so the judge does not inherit the retriever's confidence in its own find.
Core Claim
Proactive systems such as PRISM decide to intervene when a calibrated probability that the user will accept help, $p_{accept}$, exceeds a cost-derived threshold $\tau$. That probability is computed from the interaction context alone. We claim that once the assistant retrieves from a noisy store, acceptance depends on the content being pushed, and the gate should estimate
$$ p_{accept}(c, d) > \tau, $$
where $c$ is the user's interaction context up to now and $d$ is the specific retrieved item the assistant would show. A context-only gate cannot, by construction, change its decision when the gold document is swapped for a near-miss. An evidence-aware gate can, but only if training forces it to attend to $d$. With ordinary negatives, which are random moments or random documents, a model can tell positives from negatives using coarse cues and never learns the fine boundary. This is the "lazy feature" failure that hard negative mining exists to fix.
The training signal we propose therefore has two kinds of hard negatives built as counterfactual pairs around each positive $(c^*, d^*)$. A temporal hard negative keeps $d^*$ and moves the moment to a nearby turn or event before the need had formed, giving $(c^-, d^*)$. An evidence hard negative keeps $c^*$ and replaces $d^*$ with a near-miss distractor, giving $(c^*, d^-)$. A gate that ranks the positive above both members of the pair has learned to answer two separate questions: "is it time?" and "is this the thing?"
How This Relates to Prior Work
Three papers motivate the gap. ProCIS benchmarks proactive retrieval in Reddit threads that link Wikipedia articles. It scores systems with npDCG, a ranking metric that discounts suggestions delivered after the ideal turn. Its finding is that a language model which imagines target documents wins at reactive retrieval but loses to ColBERT once timing counts, so retrieval quality and timing are separable failure modes. CoDA-Bench shows that code agents searching sandboxes of roughly 980 files fail to find the correct data in about 20% of tasks, and gain 25–28% when handed the right files. Near-miss distractors are therefore the dominant hazard in realistic stores. PRISM separates need from acceptance but gates on context only and uses no external retrieval. We keep its cost-sensitive threshold and its policy of escalating to slower reasoning only near the margin. What we change is the input to the gate and the negatives it is trained on.
Two papers supply methods. DocReRank trains rerankers on LLM-generated hard negative queries that a page cannot answer, verified by a vision-language judge with an 8.2% false negative rate. We reuse its generate-then-verify loop to build evidence negatives without trusting unlabeled corpus neighbours, which is the false-negative contamination problem that paper documents. Stylianou et al. show that overly hard negatives can collapse triplet training, and propose a loss that only repels negatives inside a bounded similarity band. This constrains our design. Temporal negatives one turn before the ideal moment may be genuinely acceptable, so we apply repulsion only to negatives that annotators or the delay penalty mark as clearly worse.
Two lines of work act as foils. FLARE retrieves when token confidence drops and reaches 51.0 exact match on 2WikiMultihopQA, against 39.4 for single-shot retrieval. It never faces a user who pays for interruptions, so we adopt its low-confidence trigger for the silent retrieval stage only. ProactiveBench and ProPerSim learn when to stay silent from accepted and rejected trajectories, but their negatives are whatever the simulator happened to reject, not constructed near-misses. ProPerSim also retrieves only from the user's own interaction log. On role separation, Self-collaboration and MACM show that an independent tester or judge catches errors that a model reviewing its own output misses. Both also report that small models struggle to hold prompted roles. That negative result is why our judge is a separately trained small model rather than a prompted persona.
Research Program
The system has three parts. First, a cheap need watcher fires often, whenever context suggests a possible information need or the working model's confidence drops. Second, a silent retriever runs and reranks candidates without showing anything. Third, an interruption arbiter sees the context and the top candidate and makes the push-or-stay-silent call against $\tau$. The arbiter never sees the retriever's scores or its rationale, so a confidently ranked distractor gets no head start. Following PRISM, the arbiter escalates to a slower reasoning pass only when its estimate falls within a small margin of $\tau$, which keeps the added delay limited to borderline cases.
The main work is building counterfactual training pairs. On ProCIS, the turn where a Wikipedia link was shared serves as a weak positive. Temporal negatives are earlier turns of the same thread. Evidence negatives are articles that ColBERT ranks near the gold article but that were not shared, filtered by an LLM judge to remove ones that would also have been useful. We add generated near-miss descriptions in the DocReRank style. On ProactiveBench, which has no external corpus, we can build only temporal negatives, drawn from events adjacent to accepted interventions. This benchmark serves as a regression check: evidence-awareness must not cost timing accuracy.
On CoDA-Bench, we build a proactive variant. An observer process watches a working agent's trajectory and may inject one file suggestion at a step of its choosing. Evidence negatives here come naturally from the benchmark's same-schema distractor files.
The arbiter is trained with a pairwise loss. For each positive it must score above each of its two counterfactual negatives, and repulsion is bounded as in the SCT loss. We then compare this against the same arbiter trained on random negatives only, on temporal negatives only, and on evidence negatives only.
Baselines
The naive reference points are a purely reactive system that never interjects and an idle-timer trigger in the style of CodingGenie, which pushes the top retrieved item at natural pauses. The idle-timer baseline is given the same retriever and reranker as our system, so it differs only in the gate. On ProCIS, the strongest reported proactive number is ColBERT at npDCG 0.1944, with zero-shot LMGR at 0.1840 (ProCIS). Beating ColBERT by a margin larger than the variance across retrieval seeds is the first bar.
The strongest existing gating method is PRISM, a context-only cost-sensitive gate with margin-triggered slow reasoning. We reimplement it on the same backbone as our arbiter, with the same training events, the same threshold-tuning budget on a shared validation split, and the same retriever upstream. The only difference is that it does not see $d$ and does not receive our constructed negatives. Its reported gains on ProactiveBench are relative: a 22.78% reduction in false alarms and a 20.14% improvement in F1. No reported absolute F1 for PRISM was found on Lacuna. The strongest absolute F1 the file gives on ProactiveBench is Qwen2-7B-Proactive at 66.47% (ProactiveBench), and that is our anchor there.
The most important baseline for isolating our key ingredient is the identical evidence-aware arbiter trained with random and in-batch negatives. If it matches our version, then evidence-awareness alone is doing the work and the hard negatives are not.
On CoDA-Bench, the no-observer reference is the working agent alone. The file reports Mini-SWE-agent with GPT-5.5 at 61.1% execution accuracy overall and 49.6% on CoDA-HARD (CoDA-Bench), but we will rerun the agent with a budget-feasible backbone and report our own no-observer number. The oracle-file condition serves as the upper bound. Beating these means raising execution accuracy over no-observer while keeping the rate of harmful injections, meaning tasks that pass without the observer and fail with it, below a context-only observer's rate.
Why not just…?
Why not just train a better reranker and keep a context-only gate? This is the real threat. If DocReRank-style training made the top retrieved item almost always correct, the gate would not need to inspect evidence. We test this directly in a factorial design that crosses reranker training (standard vs. hard-negative) with gate type (context-only vs. evidence-aware). Our claim survives only if the evidence-aware gate still adds gains on top of the stronger reranker. That would happen because a reranker ranks candidates relative to each other, while the gate must decide whether even the best candidate is worth an interruption. If there is no added gain, we report that the right fix is upstream.
Why not just gate on the reranker's score with a threshold? This is cheap and plausible, so we include it as a baseline: interrupt when the context gate fires and the reranker score exceeds a tuned cutoff. Its weakness is that reranker scores are calibrated for relevance to a query, not for acceptance at a moment. A relevant document shown too early still scores high. Results on the temporal negatives in the swap test below will show whether this baseline confuses the two.
Why not just prompt a frontier model to act as a reluctant proactive assistant? ProactiveBench reports a 64.73% false-alarm rate for GPT-4o-mini, and finds that a fine-tuned 7B model (66.47% F1) beats GPT-4o (64.60%) and Claude-3.5-Sonnet (62.00%). Over-eagerness is therefore not solved by scale or instructions. We still run a frontier model, given the context and candidate document, under the same threshold tuning.
Why not just merge the watcher, retriever, and arbiter into one model? We run that ablation, with one model producing both the retrieval and the decision. We expect it to inherit confirmation bias toward its own find, as Self-collaboration observed for self-review. If it matches the separated design, we drop the separation and say so.
Proposed Experiments
The decisive check is an evidence-swap test. On held-out ProCIS positives and CoDA-Bench injection points, we keep the moment fixed and replace the gold item with a verified near-miss, then measure how often the gate switches from push to silent. We also run the mirror test: keep the item fixed and move the moment earlier, measuring the same switch rate. A context-only gate scores zero on the first test by construction. Our hypothesis predicts high switch rates on both tests for the counterfactually trained arbiter, and a high rate on only one test for each single-negative-type ablation. That pattern would show that each kind of negative teaches its own boundary.
The end-to-end outcome measures are three. On ProCIS, we report npDCG on the 100 annotated conversations. On ProactiveBench, we report F1 and false-alarm rate, where the goal is no regression from adding evidence. On CoDA-Bench, we report execution accuracy and harmful-injection rate, with the gap to the oracle-file condition measuring how much of the available 25–28% the observer recovers. Every comparison holds the backbone, retriever, training events, and threshold-tuning budget constant. We report the added delay at the 95th percentile and the share of decisions sent to slow reasoning, so the cost of the design is visible alongside its gains.
References
- PRISM: Festina Lente Proactivity—Risk-Sensitive, Uncertainty-Aware Deliberation for Proactive Agents
- ProCIS: A Benchmark for Proactive Retrieval in Conversations
- CoDA-Bench: Can Code Agents Handle Data-Intensive Tasks?
- DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers
- Active Retrieval Augmented Generation
- Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance
- ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation
- Self-Collaboration Code Generation via ChatGPT
- MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems
- CodingGenie: A Proactive LLM-Powered Programming Assistant