Language models that reason in many steps tend to trust what they have already said. When a retrieved document or a re-examined image contradicts an earlier step, or contradicts what the model memorized in pre-training, the model usually keeps going with the old story. This is called textual inertia: the model sticks to its own earlier text. Recent diagnostics show that models trained to "think longer" are the most stubborn. We propose to train the opposite habit directly. The training data are synthetic multi-hop puzzles, where answering requires chaining several facts. Every fact is invented, so memory cannot help, and partway through the reasoning the evidence is deliberately changed. A reward that can be checked automatically pays the model only for answers consistent with the evidence it is currently looking at.
The bet is that "the observation channel outranks my own prior text" is a general reasoning habit, like the decomposition and self-checking habits that math training is known to transfer. If it is, a model trained only on text puzzles should become less stubborn on tasks it never saw: counterfactual multi-hop QA, image-swap tests, and multi-turn visual dialogues seeded with hallucinations. If it is not, we will learn that resistance to stale context does not transfer across domains, which is also worth knowing.
Core Claim
Training with reinforcement learning (RL) against automatically checkable rewards on narrow domains teaches transferable habits. Planning-step search on math lifts coding and science scores (CPL). Math RL elicits self-reflection that carries over to logic and code (Shen et al.). Learned early-stopping on math also carries over to science QA (Self-Braking Tuning).
None of these rewards ever punishes a model for ignoring new evidence. In math there is no external evidence that can change mid-solution. We therefore expect math RL to transfer decomposition but not evidence-tracking. That would explain why long-thinking models are more brittle under context change, not less.
Our claim is that evidence-tracking needs its own training signal. That signal can be built cheaply from procedurally generated counterfactual worlds with mid-chain evidence swaps. Once trained this way, the habit should transfer across modality, from text-only training to visual re-grounding.
Where the Gap Comes From
Three lines of diagnostic work motivate the proposal.
CofCA rewrites Wikipedia facts into fictional counterparts and finds that GPT-4's exact match falls from about 70% to about 53% on 2-hop counterfactual questions. It also finds that only 36.3% of GPT-4's reasoning chains are correct. Models lean on memorized facts rather than the passage in front of them.
VISUALSWAP swaps an image mid-reasoning for a near-duplicate that differs in one critical detail. Qwen3-VL-32B-Thinking loses 48.3% accuracy versus 17.9% for its Instruct sibling. Phrases like "let me look again" barely raise attention to image tokens, while an explicit new conversation turn does.
Zhu et al. inject plausible errors into video-reasoning chains on STAR. Models self-correct in under 10% of cases and are contaminated by the injected error 60–90% of the time.
Layer-wise analysis of LLaVA shows a related failure at the level of single tokens. Correct visual predictions mutate into higher-frequency words in the deepest layers (Wang et al.). We use that analysis as a lens, not a method.
All three are diagnostic, and none trains a fix. The existing fixes fall into two groups.
The first group works at decoding time. Visual Contrastive Decoding subtracts the output distribution obtained from a noised image. Residual Visual Decoding shifts outputs toward an image-only context when dialogue history misleads. In the multi-hop agent study of Murphy et al., context-aware decoding and layer-contrast decoding were applied to ReAct agents, an agent design that alternates reasoning steps with search actions. These methods cost extra forward passes. Murphy et al. report that no decoder wins across model families, which constrains us: a fix that lives in decoding heuristics is fragile.
The second group trains agents. FrugalRAG, MMhops-R1 and SCM-GRPO use RL for retrieval, tool use, or verification. Each of them assumes the evidence is consistent with the model's memory and never changes mid-episode. None measures whether the trained behavior reduces inertia elsewhere.
The closest single idea is to run math RL and test it on CofCA. That is a useful baseline but not a proposal, because it predicts transfer from a reward that never touches the failure.
Research Program
Synthetic worlds. We build a generator of synthetic worlds. Each world is a random graph of invented entities with typed relations and attributes, for example a fictional person born in a fictional city that belongs to a fictional country with a stated founding year. Each world is rendered as a few dozen short documents searchable by a BM25 tool. Questions require 2–4 hops and are built bottom-up from single-hop questions sharing a bridge entity, following MuSiQue's composition recipe so that single-hop shortcuts fail. Because the world is known, every intermediate answer is checkable.
Episode types. Each training episode is one of four types, mixed in fixed proportions.
- Clean. No perturbation.
- Observation swap. After the model has used hop $k$'s document, the next search returns an explicitly newer record that changes that fact. The correct final answer follows the newer record.
- Self-chain error. A plausible wrong value is inserted into the model's own earlier reasoning, in the style of Zhu's LogicGraph perturbation. The retrieved documents stay correct, and the correct answer follows the documents.
- Memory-conflict. Real entity names are paired with altered facts, CofCA-style, but drawn from passages disjoint from CofCA.
The third type is what keeps the learned rule from collapsing into "trust whatever text is most recent." The rule we want is narrower: retrieved observations outrank the model's own prior text and its parametric memory. Conflicts between observations are resolved only by an explicit precedence marker.
Reward. The policy emits per-hop sub-answers in tags and a final answer. With $c_{\text{final}} \in \{0,1\}$ for final correctness, $c_j \in \{0,1\}$ for correctness of hop $j$'s sub-answer under the evidence current at that hop, and $H$ hops, the reward is
$$ R = c_{\text{final}} + \frac{\lambda}{H}\sum_{j=1}^{H} c_j , $$
where $\lambda$ trades chain faithfulness against the final answer. The hop term follows CofCA's observation that final answers are often right while chains are wrong. We optimize with Group Relative Policy Optimization (GRPO), a policy-gradient method that scores each sampled answer against the other samples for the same prompt, as used by FrugalRAG and SCM-GRPO.
Backbone. The backbone is Qwen2.5-VL-7B, the same model MMhops-R1 builds on. All training is text-only, so any change on image benchmarks is genuine cross-modal transfer.
Mechanistic analysis. After training, we repeat VISUALSWAP's attention analysis. The question is whether attention to image tokens rises after a swap in the trained model, where it does not in the base model. We also run the early-exit layer analysis of Wang et al. to see whether deep-layer "mutation" toward frequent tokens is reduced.
Baselines
Untuned backbone. The naive reference point is Qwen2.5-VL-7B run with the same ReAct-style prompt and tool access. It anchors every gain.
Math RL. The most important learned competitor is GRPO on GSM8K and MATH with the same backbone, step budget, rollout count and token budget as our method. This is the recipe whose cross-domain transfer is documented by CPL and Reasoning Vectors. It is the strongest competitor on the transfer question: if generic reasoning RL already removes inertia, our specialized signal is unnecessary. We count it as beaten only if our model reduces the swap-induced drop on VS-BENCH and the counterfactual-versus-factual gap on CofCA by a margin outside seed variance (three seeds), while matching it within two points on GSM8K.
Agentic multi-hop RL. The strongest agent-training competitor is FrugalRAG-style two-stage training, an explore step followed by GRPO with stopping rewards, on HotpotQA. FrugalRAG reports 68.47% model-based evaluation on HotpotQA with 2.05 searches (FrugalRAG). We reimplement it on our backbone at matched compute rather than comparing to that number directly. We must stay within two points on HotpotQA while beating it on the conflict benchmarks.
Decoding-time fixes. The strongest inference-time competitors are context-aware decoding for text and residual visual decoding for images. Murphy et al. report that context-aware decoding raises 2WikiMultiHopQA F1 for Qwen2-7B-Instruct from 43.1 to 51.8 (Murphy et al.). RVD raises LLaVA-1.5 accuracy in the hallucinated-conversation setting from about 15% to 33% while keeping over 90% on non-conflicting turns (Zhong et al.). We apply both to the untuned backbone and report accuracy per forward pass alongside raw accuracy.
Ablations. Two ablations isolate the key ingredient.
- Counterfactual RL without swaps. RL on the same synthetic worlds with only clean and memory-conflict episodes. This separates "invented facts force reading" from "changing facts force re-reading."
- Swap-only RL. Removing the self-chain-error episodes tests whether the model has merely learned a recency bias.
Why not just…?
Why not just run math RL and rely on known transfer? Math rewards never penalize ignoring new evidence. VISUALSWAP's finding that thinking-tuned models are roughly three times more vulnerable than instruct models suggests that long-reasoning training may worsen inertia. Our math-RL baseline settles this directly. If it closes the VS-BENCH and CofCA gaps as well as our method does, our central claim is false and we will report that.
Why not just use contrastive decoding? It doubles per-token compute in every hop of a multi-hop loop. Murphy et al. found its gains depend heavily on the model family, at only 1–2 F1 for Llama3-8B. It also cannot help when the conflict lies between the model's own chain and a later observation rather than between context and no-context. We compare at equal accuracy-per-compute and also report the combination, since the two approaches are not exclusive.
Why not just insert a conversation break before every new observation, or fold the history as AVCR does? This is the most serious threat. VISUALSWAP shows that structural turn boundaries restore visual attention. AVCR, a check-reason-fold loop from Zhu et al. that summarizes away erroneous history, gains 10–15% with no training. We run both as prompt-level baselines on every benchmark.
If turn breaks match our method on observation swaps, the residual case for training rests on two things. The first is self-chain errors, where there is no new observation to put after a break. The second is memory conflicts, where there is no history to fold. If breaks also match there, we would reframe the contribution as a cheap prompting result.
Why not just fine-tune on CofCA-style counterfactual data? That is our no-swap ablation, trained with the same RL budget. It tests whether invented facts alone suffice. We expect it to fix memory conflicts but not mid-chain inertia, which needs evidence that changes after the model has committed.
Proposed Experiments and Decisive Checks
In-domain and held-out text evaluation. We evaluate on CofCA with exact match and joint chain accuracy, split by hop count. The reference point is GPT-4's roughly 53% 2-hop counterfactual exact match and 36.3% chain accuracy. We also report MuSiQue-Ans, HotpotQA and 2WikiMultiHopQA F1 to confirm that ordinary multi-hop ability is preserved. Additionally, we build a text analogue of VISUALSWAP on MuSiQue, in which the bridge document is replaced after first use.
Cross-modal transfer: the decisive test. We run VS-BENCH and report the post-swap accuracy drop. No reported VS-BENCH result was found on Lacuna for Qwen2.5-VL-7B, so we measure the base backbone ourselves. We also run MMHalSnowball, reporting the flip rate under hallucinated context and accuracy on non-conflicting turns, and STAR under the LogicGraph perturbation, reporting the self-correction rate.
Guards against over-credulity. A model that simply flips to any new text would look good on swaps. Two measurements guard against this. The first is MMHalSnowball's non-conflicting accuracy, which must stay near the base model. The second is a held-out set where an injected observation is marked as superseded and should be ignored.
General reasoning. GSM8K and the MMhops Bridging split (MMhops-R1 reports 51.35%) check that math and multimodal multi-hop ability are not traded away.
What would count as success. The result is decisive if three things hold together. Our model shrinks the VS-BENCH swap drop and the MMHalSnowball flip rate relative to both math RL and the no-swap ablation. It does so without losing non-conflicting accuracy. And the gain survives against turn-break prompting on the self-chain-error and memory-conflict splits. The post-swap attention analysis should show image attention rising after the swap, which would tie the behavioral gain to actual re-grounding rather than better guessing.
References
- CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks
- Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
- Let LRMs Break Free from Overthinking via Self-Braking Tuning
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark
- Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination
- Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts
- Towards Understanding How Knowledge Evolves in Large Vision-Language Models
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
- An Analysis of Decoding Methods for LLM-based Agents for Faithful Multi-Hop Question Answering
- FrugalRAG: Less is More in RL Finetuning for Multi-hop Question Answering
- MMhops-R1: Multimodal Multi-hop Reasoning
- Grounding Multi-Hop Reasoning in Structural Causal Models via Group Relative Policy Optimization
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic