Introduction
Abductive reasoning—the ability to infer the most likely explanation for an observation—is a cornerstone of human intelligence. While Vision-Language Models (VLMs) have made significant strides in descriptive tasks, their capacity for causal and narrative inference remains under-explored. This paper introduces NL-Eye, a novel benchmark designed to evaluate visual abductive reasoning.
Current VLM evaluations often focus on "what" is in an image, but real-world applications require understanding "why." For instance, seeing a person with a cast (premise) and inferring they likely slipped on a wet floor lacking a warning sign (plausible hypothesis) requires connecting visual cues across scenes to form a logical narrative. Without this skill, AI cannot reliably predict accidents or verify the logic of generated content.
The NL-Eye Benchmark
The NL-Eye dataset consists of 350 manually curated triplets, totaling 1,050 images. These examples cover six reasoning categories: physical, functional, logical, emotional, cultural, and social. To ensure high quality and visual consistency, the images were generated using advanced text-to-image models (Midjourney and DALL-E 3) with human-in-the-loop intervention, such as inpainting and seed-matching, to maintain character and setting continuity across the premise and hypotheses.
Methodology
The benchmark employs two primary evaluation frameworks:
- The Triplet Setup: Models are presented with a premise and two hypotheses. They must identify the more plausible hypothesis and provide a reasoning explanation. To mitigate positional bias, the order of hypotheses is swapped, and only consistent correct answers are credited.
- The Pairs Setup: Models assign a plausibility score (1-10) to a single premise-hypothesis pair.
To evaluate the quality of AI-generated explanations, researchers utilized both human annotators and an "LLM-as-a-judge" (GPT-4o) to determine if the model's logic aligned with human intuition.
Key Findings: The Recognition Gap
The study reveals a significant "recognition gap" in current VLMs. Interestingly, when models were provided with textual descriptions of the scenes instead of images, their reasoning performance shot up to approximately 80%. This suggests that the primary bottleneck is not logical reasoning itself, but rather visual encoding—the ability to detect and interpret the significance of subtle visual details, such as a missing wedding ring or a specific safety sign.
Performance Comparison
- Human Baseline: Humans achieve 85% accuracy in prediction and 95% in explanation quality.
- State-of-the-Art VLMs: GPT-4o leads the field with 60% consistency accuracy but remains 25 points behind human performance.
- Open-Source Models: Models like LLaVA 1.6 and Fuyu struggle significantly, often performing near or below random chance (4% to 14% accuracy).
Conclusion
NL-Eye highlights a critical deficiency in current AI: the inability to perform intuitive abductive reasoning from visual data. While closed-source models like GPT-4o show promise, the gap between human and machine performance remains vast. Future research must focus on improving visual interpretation to enable models to build more robust "mental models" of the physical and social world, moving beyond simple pattern matching.
Summary updated