Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning

Chen Ling, Hanqian Li, Dongnan Liu, Keyu Qian, Jungang Li, Xinglong liu, Shiyi Wang, Xin Dong, Pengcheng Zhu, Wei Zhou, Linjian Mo, Nai DingarXiv 2026 · arXiv

TL;DR

Multimodal Large Language Models (MLLMs) frequently fail at counter-commonsense reasoning—like recognizing a chair floating in mid-air—because their language decoders override visual evidence with strong text-based priors. This paper introduces TACT (Text-Anchored Cross-Modal Transfer), a post-training framework that uses purely text-only counterfactual data to debias the decoder. Without requiring any new visual training data, TACT significantly boosts visual counter-commonsense reasoning while fully preserving general model capabilities.

What Problem Does This Paper Address?

When presented with surreal or counter-commonsense images, state-of-the-art MLLMs often hallucinate a normal scene, choosing what they expect to see over what is actually there. Most existing mitigation strategies assume this is a visual grounding or perception failure, attempting to fix it by enhancing the visual inputs or vision encoders.

The authors challenge this assumption. Through multi-trajectory decoding analysis, they reveal that MLLMs do perceive the anomalous visual evidence correctly, and the right answer exists within their decoding space. The true bottleneck is a decision-level prior bias located in the shared language decoder: when visual evidence conflicts with high-frequency language expectations, the decoder simply overrides the eyes.

An example of an MLLM fabricating grounding evidence to match its commonsense prior (Bad Trajectory) versus correctly identifying the anomaly (Good Trajectory).

Methods

To correct this decoder-level policy bias without destroying the model's general knowledge, the authors developed a text-only intervention consisting of two main parts:

1. Text-Anchored Data Construction Pipeline

The core component here is Factuality-Frequency Distillation (FFD). FFD probes the target MLLM using text-only queries to estimate how strongly it holds specific commonsense beliefs (e.g., "strawberries are red"). If a prior is strongly entrenched, the pipeline substitutes a rare or counterfactual value ("blue strawberry") and uses an LLM to synthesize a high-quality text description and corresponding QA pairs (True/False, Multiple Choice, or Binary Choice).

Overview of the three-step text-anchored data construction pipeline.

2. Text-Anchored Cross-Modal Transfer (TACT)

Using the generated text corpus, TACT fine-tunes the shared language decoder (keeping the vision encoder and projector completely frozen) via a two-stage post-training strategy:

  • Stage 1: On-Policy Trajectory Distillation: Uses Supervised Fine-Tuning (SFT) on curated, faithful reasoning paths to teach the model to default to evidence-following behavior.
  • Stage 2: Faithful-over-Prior Preference Optimization: Employs Direct Preference Optimization (DPO) to explicitly train the model to prefer evidence-following trajectories over prior-driven ones under the same prompt.

Datasets

  • Training Data: A newly synthesized text-only corpus containing 7,830 counter-commonsense items across 6 major categories (Agency, World Facts, Physical Relations, Magnitude, Appearance, Social Roles). The underlying facts were mined from Visual Genome, Wikidata, and Distributions-over-Quantities (DoQ).
  • Evaluation Benchmarks: Evaluated on three prominent counter-commonsense visual datasets: CDH-Bench (MC and QA splits), CAIT, and VLind-Bench.
  • Capability Retention Benchmarks: To ensure the model didn't lose its standard abilities, it was tested on general VQA benchmarks (MMBench-en/zh, HallusionBench) and standard commonsense subsets (CDH-CS, VLind-CS).

Key Insights and Ideas

  • Perception is Not the Bottleneck: By increasing the sampling budget ($k$) during decoding, the authors showed that correct answers surface naturally as the language prior's grip weakens. This proves the visual features are present; they are just ignored during greedy decoding.
  • Cross-Modal Transfer via Text Only: Because the bias is fundamentally a linguistic decision policy, training the model on text-only counterfactual descriptions successfully transfers to visual counterfactual reasoning tasks. The model learns a general rule: "trust the explicit evidence provided in the context over your internal defaults."

Key Results

  • Massive Accuracy Gains: TACT (built on a Qwen3-VL-8B backbone) achieved an average accuracy of 80.4% across the counter-commonsense benchmarks, a massive leap from the base model's 66.8%. It even comfortably beat the much larger Qwen3-VL-32B model (71.3%).
  • Reduced Prior Bias: The average prior-bias metric dropped significantly from 31.5% to 18.2%.
  • Zero Degradation on Standard Tasks: TACT fully preserved the model's performance on general VQA and standard commonsense tasks, with fluctuations staying within a negligible $\pm0.3$ accuracy points.
Qualitative examples showing TACT successfully resisting dominant language priors to answer correctly based on visual anomalies.

Limitations and Future Work

The paper does not explicitly dedicate a section to limitations, but a few inherent constraints are apparent:

  • The framework relies heavily on a powerful teacher model (like Qwen3.5-397B) to curate high-quality reasoning trajectories for distillation.
  • The data generation pipeline depends on structured knowledge bases and predefined taxonomies, which might limit its coverage of highly niche or evolving real-world commonsense facts.
  • Future work could explore expanding this text-anchored debiasing technique to other forms of MLLM hallucinations beyond counter-commonsense conflicts.

Summary updated

Abstract

The visual reasoning ability of multimodal large language models (MLLMs) is crucial for downstream applications, particularly counter-commonsense reasoning, which requires models to reason beyond common assumptions. Recent studies mainly improve visual counter-commonsense reasoning by enhancing visual inputs, following the assumption that failures originate from insufficient visual grounding. However, our empirical analysis reveals that the bottleneck is not visual perception. MLLMs already capture the relevant visual evidence, and the correct answer exists in their decoding space. Instead, the shared language decoder resolves prior--evidence conflicts by favoring dominant language priors, especially for low-frequency factual scenarios. Motivated by this, we first propose a text-anchored data construction pipeline, whose core component, Fact-Frequency Distillation (FFD), estimates the prior strength of commonsense facts and distills verified counter-commonsense scenarios into a high-quality text corpus. Building upon this corpus, we introduce TACT, a text-anchored post-training framework that debiases the shared language decoder without requiring any visual training data. TACT routes evidence-following and prior-driven reasoning trajectories into different optimization stages, enabling the decoder to resolve prior--evidence conflicts. Across counter-commonsense visual benchmarks, TACT substantially improves visual reasoning while preserving general capabilities, demonstrating effective text-to-vision cross-modal transfer.
BibTeX
@misc{ling2026debias,
  author = {Chen Ling and Hanqian Li and Dongnan Liu and Keyu Qian and Jungang Li and Xinglong liu and Shiyi Wang and Xin Dong and Pengcheng Zhu and Wei Zhou and Linjian Mo and Nai Ding},
  title = {{Debias in Text, Believe Your Eyes: Text-Anchored Cross-Modal Transfer for Visual Counter-Commonsense Reasoning}},
  year = {2026},
  eprint = {2608.06938},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.06938},
}
Figure