When a reasoning model is handed a misleading hint, thinking harder makes things worse. It writes a longer chain of thought that bends the evidence until it fits the false premise. On SciAux, two misleading snippets drop Qwen3-8B from 96.22% to 77.03% with thinking enabled, which is below its 74.87% with thinking off (Thinking in a Crowd). The usual way to get more out of a model at test time is to sample several answers and vote (self-consistency). That fails here because every sample sees the same poisoned context. The samples differ in wording but share the false premise. In quality-diversity terms, they are diverse on the surface and identical in substance. Turning up the temperature to get real diversity damages fluency and logic (Hugh Zhang et al.; Jiacheng Xu et al.).
We propose moving diversity out of the decoder and into the input. The model runs several parallel reasoning branches, each decoded at low temperature. Each branch sees the full context minus a different group of evidence: a text snippet, or a coherent cluster of projected visual tokens such as one object region. The groups dropped by different branches are chosen to be maximally dissimilar from one another. Comparing answers across branches then shows which evidence group flips the answer. The final answer is produced without the groups that flip it. The model's reasoning ability is untouched. The method only keeps the model from spending that ability on rationalizing a distractor. This matches the main lesson of the base-model-bottleneck literature: scaffolding can release capability the model already has but cannot create new capability.
Core Claim
The claim has two parts. First, under misleading context, answers from ordinary sampled self-consistency are highly correlated because the poisoned premise is present in every sample. Branches that see different subsets of evidence break that correlation, and they do so without the degeneration that high-temperature sampling causes. Second, the gain is capped by the model's own clean-context accuracy. The method should close part of the gap between misleading-context accuracy and clean accuracy, and should never exceed clean accuracy. This prediction makes the work a test of the bottleneck view as well as a method.
The new element is not parallel branching, which Superposition Prompting already does for retrieved documents, and not diversity-based token selection, which DivPrune already does. It is two things together. One is using leave-group-out evidence diversity across branches as the source of answer diversity. The other is applying it at the boundary where projected visual tokens enter the language model, which is where rationalization of misleading visual cues has not been studied.
Research Program
Evidence grouping
The input is split into evidence groups. For text, a group is an auxiliary snippet or retrieved passage. For images and video, a group is a cluster of visual tokens after the projector. Clusters are formed by agglomerating tokens by cosine similarity in the projected space, which is the same distance DivPrune uses, so that a coherent distractor region stays in one group rather than being spread across many.
Choosing which groups each branch drops
For $K$ branches, we choose $K$ drop-sets. Two constraints apply. Each drop-set removes at most a fraction $\rho$ of the evidence, so that every branch still has enough to answer. Across branches, the drop-sets are chosen by a greedy max-min diversity rule, so that no two branches remove similar material. The query and system instruction are always kept. This reuses DivPrune's max-min selection, but applied to sets of groups that are removed, not to tokens that are kept.
Aggregation and attribution
Each branch decodes once, at low temperature, with thinking enabled. Let $p_k(a)$ be branch $k$'s probability for answer $a$ (or its vote), and let $D_k$ be its drop-set. The influence of group $g$ is
$$ \Delta_g = \Big\| \tfrac{1}{|S_g|}\textstyle\sum_{k \in S_g} p_k - \tfrac{1}{|\bar S_g|}\sum_{k \in \bar S_g} p_k \Big\|_1 , $$
where $S_g$ is the set of branches that saw $g$ and $\bar S_g$ is the set that did not. Groups with large $\Delta_g$ are evidence the answer depends on. That evidence is either essential or misleading.
To tell the two apart, we exploit the rationalization signature reported by Thinking in a Crowd: misleading context inflates reasoning length. The rule has three steps. First, flag a high-$\Delta_g$ group as suspect when the branches that saw it produce markedly longer chains and disagree with the majority of branches that did not. Second, run one final pass with the suspect groups removed. Third, if no group is flagged, take the majority vote over the branches.
We will report how often this length-based heuristic misfires, because it is the weakest assumption in the design.
Settings
The text setting uses SciAux (dataset), which has helpful, irrelevant, and misleading snippet conditions over 6,828 ScienceQA-derived questions.
For the multimodal setting, we build a visual counterpart. SciAux comes from ScienceQA, and many of those items have images, so we take the image-bearing subset. We add misleading visual evidence in two ways: rendered misleading captions or labels pasted into the image, and distractor frames spliced into NExT-QA videos.
For each setting, we test an open Qwen3-based model with switchable thinking at two or more sizes, so that the predicted ceiling can be measured against clean accuracy.
Baselines
The naive reference is a single pass with thinking on, compared with thinking off, on the full noisy context. On SciAux with two misleading snippets and Qwen3-8B, these are 77.03% and 74.87% respectively, against a clean ceiling of 96.22% (Thinking in a Crowd). We rerun these numbers ourselves to calibrate. Beating this reference means a statistically reliable gain over 77.03% in that cell, with no more than a one-point loss in the clean and helpful conditions.
The strongest competitor on our exact question is self-consistency with temperature sampling at the same number of forward passes $K$. No reported result found on Lacuna for self-consistency on SciAux, so we establish this baseline ourselves. Its temperature is swept on a held-out split with the same tuning budget we give to $\rho$ and $K$. It is the right comparison because it isolates the key question: does diversity in the input beat diversity in the output at equal compute? Beating it means higher accuracy in the misleading conditions at every matched $K$ from 3 to 16.
The strongest existing noise-robust method is Ext2Gen, which trains a model with preference optimization to extract key sentences before answering. On its RAG suite it reports an average score of 0.557 against 0.473 for the default Llama3.1-8b-instruct (Ext2Gen). We train it on the same base model and data budget and evaluate it on SciAux. Our method needs no training, so matching Ext2Gen counts as a practical win, and combining the two is a natural follow-up.
On the visual side, we compare against Superposition Prompting applied to text snippets. It reports a 43% accuracy gain over naive RAG on NaturalQuestions-Open with MPT-7B, and it isolates each document in its own branch but never drops evidence in a structured way. We also compare against Vittle, which adds a learned compression layer inside the model to filter perturbations and reports up to +1.6% on POPE under corruptions.
Two ablations isolate the key ingredient. The first replaces max-min drop-sets with random drop-sets of the same size. The second removes attribution and simply votes over the evidence-subset branches.
Why not just…?
Why not just sample more answers at higher temperature? Because a misleading premise in the shared context should make sampled chains fail together. More samples then reinforce the error rather than average it out, and pushing temperature high enough to escape the premise brings the degeneration documented by Tianhui Zhang et al.. This is the real threat to the proposal. We settle it directly in two ways. We measure the error correlation between pairs of branches under each scheme. We also plot the matched-compute accuracy curve. If self-consistency closes as much of the gap, the core claim is false and we will report it as such.
Why not just turn thinking off, or tell the model to ignore irrelevant context? Turning thinking off avoids the trap only by giving up the 10–13% gain that thinking provides on clean inputs (Thinking in a Crowd), and it still scores 74.87% under misleading snippets. A prompt instruction to ignore irrelevant text is cheap, so we include it as an extra baseline. We expect it to help little, because the model cannot tell which snippet is misleading. That is exactly the information the cross-branch comparison provides.
Why not just prune visual tokens at the projector with DivPrune or a bottleneck? These methods remove redundant or low-level noise. A misleading cue, however, is usually distinctive, so diversity-based selection tends to keep it. Learned bottlenecks also cost fine-grained detail such as text in images, as Vittle itself notes. We run DivPrune and Vittle on the visual misleading set and report OCR-heavy items separately to test both points.
Proposed Experiments
The first decisive check is the main SciAux grid: three conditions, crossed with thinking on and off, crossed with $K$ in {3, 5, 8, 16}, comparing our method against matched self-consistency. Here we also measure error correlation between pairs of branches.
The second check tests the bottleneck prediction. Across model sizes, we regress the recovered gap (method accuracy minus noisy accuracy) on the clean-noisy gap. Recovery should stay a fraction of that gap and never exceed clean accuracy. A small model that is wrong on clean inputs should gain nothing. This mirrors the observation in Chain-of-Glimpse that 3B and 7B models track the right objects yet misread intent.
The third check transfers the method to the visual misleading SciAux subset, POPE under corruptions, and NExT-QA with spliced distractor frames. On NExT-QA, the reference is CoG-7B's 83.3% against a 79.7% baseline, measured on the clean benchmark.
The fourth check examines attribution quality. On the constructed sets we know which group is misleading, so we report precision and recall of $\Delta_g$-based flagging and how often the length heuristic wrongly removes helpful evidence.
The fifth check is a cost accounting. Branches share a key-value cache for the common prefix, as in Superposition Prompting, so we report wall-clock time against self-consistency, not only the number of forward passes.
References
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Unifying Human and Statistical Evaluation for Natural Language Generation
- Massive-scale Decoding for Text Generation using Lattices
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Visual Instruction Bottleneck Tuning
- Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning
- Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understanding