Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs

Jiaao Yu, Shenwei Li, Mingjie Han, Yifei Yin, Wenzheng Song, Chenghao Jia, Man Lan

TL;DR

Researchers have introduced MPCC (Masked Prediction via Context and Commonsense), a novel fine-tuning task that forces Vision-Language Models (VLMs) to infer missing objects in an image using logic and world knowledge. By combining this task with a new training strategy called Reinforcement Fine-Tuning (RFT) with Prior Sampling, they significantly boosted the reasoning and generalization capabilities of models like Qwen2.5-VL, outperforming standard perception-based training.

What Problem Does This Paper Address?

While Large Language Models (LLMs) have made massive leaps in reasoning (think DeepSeek-R1 or OpenAI’s o1), Vision-Language Models (VLMs) are lagging behind. Most current VLMs excel at perception—identifying what is clearly visible—but struggle with reasoning—understanding the "why" or "what if" of a scene.

A major hurdle is that visual reasoning lacks the "verifiable rewards" found in math or coding tasks. In math, an answer is either right or wrong; in vision, describing a scene is subjective. Current attempts to fix this often reduce images to text summaries or focus on simple detection, which fails to activate the model's inherent commonsense. The authors argue that to reach Artificial General Intelligence (AGI), VLMs must be able to "fill in the blanks" of a scene using context, much like the human brain does.

Methods

The authors propose two major innovations:

  1. The MPCC Task: Instead of asking a model to describe an image, they mask a key entity (e.g., covering a fire hydrant with a black box) and ask the model to predict what is hidden. To succeed, the model must use visual context (e.g., "there is a hose nearby") and commonsense (e.g., "fire hydrants are usually on sidewalks near roads").
  2. RFT with Prior Sampling: Building on Group Relative Policy Optimization (GRPO), the authors introduced a hybrid training approach. Standard Reinforcement Learning (RL) can be inefficient, while Supervised Fine-Tuning (SFT) often leads to "cognitive rigidity" (overfitting). Their method replaces one model-generated sample in an RL group with a high-quality, human-annotated reasoning trace ("prior sampling"). This guides the model toward better reasoning paths without the downsides of pure supervision.

Datasets

  • MPCC-Eval (New): A specialized benchmark created from the COCO dataset. it contains 1,114 image-mask pairs categorized into three difficulty levels: Easy, Moderate, and Hard.
  • COCO: Used as the base for generating training and evaluation data.
  • BLINK: Used for "cross-task generalization" testing, specifically focusing on Spatial Relation, Semantic Correspondence, Relative Depth, and Object Localization.

Key Insights and Ideas

  • Masked Prediction as a Reasoning Engine: Inspired by Gestalt psychology, the authors treat image semantics as a holistic system. Masking forces the model into "cross-modal compensatory reasoning," where it must use language-based commonsense to solve a visual puzzle.
  • The "SFT vs. RFT" Trade-off: The study found that while SFT (training on specific examples) gives a quick performance boost on known data, it fails to generalize. RFT with Prior Sampling acts as a middle ground, maintaining high performance while significantly improving Out-of-Distribution (OOD) generalization.
  • Reasoning Improves Perception: Interestingly, training a model to reason about hidden objects actually makes it better at other downstream tasks like determining "Relative Depth" or "Spatial Relations," even though it wasn't specifically trained for them.

Key Results

  • Performance Boost: On the MPCC-Eval benchmark, fine-tuned models showed dramatic improvements. For example, a Qwen2.5-VL-3B model improved its "Hard" task score from 21.43 to 58.85 after SFT.
  • Generalization: In Out-of-Distribution tests (testing on object categories the model never saw during training), the RFT with Prior Sampling method achieved a score of 74.20, significantly higher than the base model's 61.74.
  • Qualitative Success: The authors showed that models trained with MPCC use more "reasoning-heavy" vocabulary in their internal chain-of-thought, using words like "positioned," "nearby," "size," and "reasonable" to justify their answers.

Limitations and Future Work

The authors acknowledge that while MPCC is effective, scaling this to massive, open-world datasets remains a challenge. Currently, the "verifiable rewards" rely on a multiple-choice format or exact string matching, which can be restrictive.

Future directions include: * Scaling the MPCC task to larger-scale training sets. * Developing more flexible verifiable reward signals that don't rely on fixed multiple-choice options. * Exploring how this reasoning "activation" affects even larger models (70B+ parameters).

Summary updated

BibTeX
@inproceedings{yu2026activating,
  author = {Jiaao Yu and Shenwei Li and Mingjie Han and Yifei Yin and Wenzheng Song and Chenghao Jia and Man Lan},
  title = {{Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i33.40019},
}
Figure