Abstract
Recent advancements in Vision-Language Models (VLMs) have enabled sophisticated multi-modal reasoning. However, these models often struggle with complex multiple-choice questions where distractors can lead to incorrect inferences. We present MM-PoE (Multi-Modal Process of Elimination), a model-agnostic two-step reasoning framework that mimics human test-taking strategies. By first identifying and masking implausible distractors before making a final selection, MM-PoE significantly improves accuracy across complex visual reasoning benchmarks like ScienceQA and AI2D in both zero-shot and few-shot settings.
Introduction
When humans encounter difficult multiple-choice questions, we rarely evaluate all options with equal weight. Instead, we typically employ a "process of elimination," crossing out obviously incorrect answers to focus cognitive resources on the remaining plausible candidates. This heuristic is particularly effective in visually complex tasks, such as interpreting scientific diagrams or maps, where subtle details distinguish the correct answer from clever distractors.
Current Vision-Language Models (VLMs) generally lack this structured approach. Most standard evaluation methods involve calculating the likelihood of each option independently or processing the entire list of choices simultaneously. This often results in the model being "distracted" by superficially similar but incorrect options. While Process of Elimination (PoE) strategies have been explored for text-only Large Language Models (LLMs), their adaptation to the multi-modal domain—and specifically to few-shot learning scenarios—has remained an open challenge until now.
Methodology: The MM-PoE Framework
MM-PoE introduces a dual-step scoring paradigm designed to narrow the model's focus. This approach is model-agnostic, allowing it to be applied to existing VLM architectures such as GIT or BLIP without the need for additional training or fine-tuning.
Step 1: Elimination
In the first phase, the VLM calculates a plausibility score for every available option based on the provided image and question. The framework then determines a threshold—typically the average score of all choices. Any option scoring below this threshold is deemed "implausible" and flagged for elimination.
Step 2: Prediction
Once the distractors are identified, the model is presented with a modified prompt. The eliminated options are either replaced with a [MASK] tag or explicitly labeled as options to be ignored. The VLM then re-evaluates only the remaining "surviving" options to make its final selection. This two-step process ensures that the final decision is made from a refined pool of plausible candidates, reducing the interference of noise from incorrect distractors.
Experimental Setup
The researchers evaluated MM-PoE on two primary multi-modal benchmarks, standardized to questions with four options:
- ScienceQA: A large-scale dataset featuring multi-modal science questions across various subjects and grade levels.
- AI2D (Diagram Understanding): A specialized dataset focused on schematic information, such as food chains or mechanical diagrams, requiring precise visual-logical reasoning.
The study compared MM-PoE against several baseline scoring methods, including standard Language Modeling (LM) likelihood, length-normalized averages (AVG), and calibration techniques.
Key Results and Insights
The experimental results demonstrate that MM-PoE consistently outperforms or matches common baseline scoring methods across different models and datasets.
Performance Gains
A notable performance jump was observed using the git-base-textvqa model on the ScienceQA dataset. MM-PoE reached 28.2% accuracy, a significant improvement over the standard LM baseline of 21.8%. Similar trends were observed in the AI2D dataset, where the structured elimination process helped the model navigate complex diagrammatic questions more effectively.
Robustness in Few-Shot Learning
Unlike previous PoE research which was largely confined to zero-shot settings, MM-PoE maintains its effectiveness in 1-shot and 3-shot scenarios. This suggests that the strategy is robust and can successfully integrate with provided context and examples, further enhancing the model's reasoning capabilities.
The Importance of Contextual Focus
The core insight of this work is that structured reasoning—how a model processes information—can be as critical as model scale. By forcing the model to "narrow its gaze" through masking, the attention mechanism can more accurately weigh subtle differences between plausible answers without the "noise" of obvious distractors.
Discussion and Future Work
While MM-PoE shows significant promise, the authors identified several areas for further refinement. One challenge is the "residual" problem: even when options are masked, the model's internal representations may still be influenced by the presence of the masked text. Future work could explore "hard deletion"—completely removing the text of eliminated options—to further reduce interference.
Additionally, the current "below average" threshold for elimination is a simple heuristic. Investigating more sophisticated, learnable scoring functions could lead to even more precise elimination. Finally, applying MM-PoE to high-stakes domains like medical imaging or professional scientific classification represents an exciting frontier for this "art of elimination."
Summary updated