TL;DR
Multimodal Large Language Models (MLLMs) frequently fail at counter-commonsense reasoning—like recognizing a chair floating in mid-air—because their language decoders override visual evidence with strong text-based priors. This paper introduces TACT (Text-Anchored Cross-Modal Transfer), a post-training framework that uses purely text-only counterfactual data to debias the decoder. Without requiring any new visual training data, TACT significantly boosts visual counter-commonsense reasoning while fully preserving general model capabilities.
What Problem Does This Paper Address?
When presented with surreal or counter-commonsense images, state-of-the-art MLLMs often hallucinate a normal scene, choosing what they expect to see over what is actually there. Most existing mitigation strategies assume this is a visual grounding or perception failure, attempting to fix it by enhancing the visual inputs or vision encoders.
The authors challenge this assumption. Through multi-trajectory decoding analysis, they reveal that MLLMs do perceive the anomalous visual evidence correctly, and the right answer exists within their decoding space. The true bottleneck is a decision-level prior bias located in the shared language decoder: when visual evidence conflicts with high-frequency language expectations, the decoder simply overrides the eyes.
Methods
To correct this decoder-level policy bias without destroying the model's general knowledge, the authors developed a text-only intervention consisting of two main parts:
1. Text-Anchored Data Construction Pipeline
The core component here is Factuality-Frequency Distillation (FFD). FFD probes the target MLLM using text-only queries to estimate how strongly it holds specific commonsense beliefs (e.g., "strawberries are red"). If a prior is strongly entrenched, the pipeline substitutes a rare or counterfactual value ("blue strawberry") and uses an LLM to synthesize a high-quality text description and corresponding QA pairs (True/False, Multiple Choice, or Binary Choice).
2. Text-Anchored Cross-Modal Transfer (TACT)
Using the generated text corpus, TACT fine-tunes the shared language decoder (keeping the vision encoder and projector completely frozen) via a two-stage post-training strategy:
- Stage 1: On-Policy Trajectory Distillation: Uses Supervised Fine-Tuning (SFT) on curated, faithful reasoning paths to teach the model to default to evidence-following behavior.
- Stage 2: Faithful-over-Prior Preference Optimization: Employs Direct Preference Optimization (DPO) to explicitly train the model to prefer evidence-following trajectories over prior-driven ones under the same prompt.
Datasets
- Training Data: A newly synthesized text-only corpus containing 7,830 counter-commonsense items across 6 major categories (Agency, World Facts, Physical Relations, Magnitude, Appearance, Social Roles). The underlying facts were mined from Visual Genome, Wikidata, and Distributions-over-Quantities (DoQ).
- Evaluation Benchmarks: Evaluated on three prominent counter-commonsense visual datasets: CDH-Bench (MC and QA splits), CAIT, and VLind-Bench.
- Capability Retention Benchmarks: To ensure the model didn't lose its standard abilities, it was tested on general VQA benchmarks (MMBench-en/zh, HallusionBench) and standard commonsense subsets (CDH-CS, VLind-CS).
Key Insights and Ideas
- Perception is Not the Bottleneck: By increasing the sampling budget ($k$) during decoding, the authors showed that correct answers surface naturally as the language prior's grip weakens. This proves the visual features are present; they are just ignored during greedy decoding.
- Cross-Modal Transfer via Text Only: Because the bias is fundamentally a linguistic decision policy, training the model on text-only counterfactual descriptions successfully transfers to visual counterfactual reasoning tasks. The model learns a general rule: "trust the explicit evidence provided in the context over your internal defaults."
Key Results
- Massive Accuracy Gains: TACT (built on a Qwen3-VL-8B backbone) achieved an average accuracy of 80.4% across the counter-commonsense benchmarks, a massive leap from the base model's 66.8%. It even comfortably beat the much larger Qwen3-VL-32B model (71.3%).
- Reduced Prior Bias: The average prior-bias metric dropped significantly from 31.5% to 18.2%.
- Zero Degradation on Standard Tasks: TACT fully preserved the model's performance on general VQA and standard commonsense tasks, with fluctuations staying within a negligible $\pm0.3$ accuracy points.
Limitations and Future Work
The paper does not explicitly dedicate a section to limitations, but a few inherent constraints are apparent:
- The framework relies heavily on a powerful teacher model (like Qwen3.5-397B) to curate high-quality reasoning trajectories for distillation.
- The data generation pipeline depends on structured knowledge bases and predefined taxonomies, which might limit its coverage of highly niche or evolving real-world commonsense facts.
- Future work could explore expanding this text-anchored debiasing technique to other forms of MLLM hallucinations beyond counter-commonsense conflicts.
Summary updated