TL;DR
While state-of-the-art Multi-Modal Large Language Models (MLLMs) like GPT-o3 can easily ace complex visual tasks like GeoGuessr, their performance plummets to near-zero when tasked with finding subtle, context-dependent anomalies. This paper introduces CaughtCheating, a novel benchmark inspired by social media sleuths trying to identify hidden clues that contradict a partner's claim (e.g., finding a second person's reflection in a spoon when someone claims to be dining alone). The findings reveal a critical gap in MLLMs' "detective-level" visual perception and active exploration capabilities.
What Problem Does This Paper Address?
Recent frontier MLLMs have achieved near-ceiling scores on standard vision-language benchmarks. They can even out-perform most humans on highly specialized tasks, such as pinpointing geographic locations from obscure visual cues or finding expertly camouflaged animals.
However, the authors identify a major blind spot: detective-level visual reasoning. Excellent human detectives excel at noticing minuscule, low-salience cues and weaving them into situational, socially grounded explanations. Current benchmarks fail to test this because they explicitly define the target object in the prompt. This paper investigates whether MLLMs can actively explore an image to find unguided, context-dependent anomalies that violate a specific social claim.
Methods
Rather than introducing a new architecture, the authors establish a rigorous evaluation framework grounded in cognitive science—specifically, Guided Search Theory. This theory states that visual attention is driven by factors like bottom-up salience (how much an item "pops out") and top-down feature guidance (knowing what features to look for).
To diagnose exactly why models fail, the authors implement a Decomposition Analysis. For every image, they generate two types of sub-questions:
- Decomposed Perception Questions: Test if the model can see the clue when explicitly told where to look (e.g., "Is there a pair of slippers under the table?").
- Decomposed Reasoning Questions: Test if the model understands the social implication of the clue (e.g., "Does the presence of these slippers contradict the claim of being alone?").
This allows researchers to isolate whether a model fails because it is visually blind to the clue, or because it simply doesn't understand its contextual significance.
Datasets
The authors introduce the CaughtCheating benchmark, consisting of 100 real-world images sourced from public social media posts. The dataset is split evenly:
- Clued Category (50 images): Contains a definitive, deterministic clue that violates a stated claim, along with weaker non-deterministic clues.
- Unclued Category (50 images): Innocent images containing no evidence of deception.
The images primarily feature common real-world settings: hotels (69%), dining venues (29%), and karaoke bars (2%). The authors manually screened the data and applied minimal cropping to preserve subtle indicators of another person's presence while obscuring obvious giveaways. Notably, they explicitly avoided synthetic AI-generated images because current text-to-image models make hidden clues far too overt and obvious to detect.
Key Insights and Ideas
- The Failure of Exhaustive Grid Search: Advanced models like GPT-o3 approach visual reasoning via a methodical, quadrant-by-quadrant grid search. While highly effective for finding Waldo, this strategy fails in real-world contexts where the target object isn't predefined. Lacking top-down feature guidance, the model simply doesn't know what constitutes an anomaly.
- The "Accuse Everything" Bias: When evaluating innocent images (the Unclued category), advanced reasoning models suffer from severe overthinking and hallucination. Instead of concluding that there is no evidence, they tend to flag benign details—like the amount of food or the number of cups—as highly suspicious.
- Perception is Not the Bottleneck: The decomposition analysis revealed that models achieve high accuracy on Perception questions when directed to the clue. The true bottleneck is the active exploration strategy—knowing where to look and how to connect a localized detail back to the overarching social claim.
- Safety Alignment Interference: Interestingly, proprietary models like GPT-4o and GPT-o3 saw degraded reasoning scores because their safety filters caused them to refuse to answer gender-related questions, which are often central to relationship-based context clues.
Key Results
The benchmark proved to be incredibly challenging for even the most advanced models:
- GPT-o3, the top-performing model, achieved an overall F1 score of just 23.9%.
- On the Clued category, GPT-o3 managed 26.0% accuracy in detecting deterministic clues, with an Intersection over Union (IoU) of 17.2%.
- On the Unclued category, GPT-o3’s accuracy dropped to a dismal 8.0%, driven by its tendency to hallucinate suspicion in innocent photos.
- In comparison, human participants achieved a 59.6% overall F1 score (56.0% Clued Accuracy and 68.0% Unclued Accuracy), vastly outperforming all agentic MLLMs.
Limitations and Future Work
The authors acknowledge several limitations tied to data scarcity:
- Sample Size: The dataset is relatively small (100 images) due to the difficulty of sourcing and verifying real-world examples.
- Demographic Bias: Because the dataset relies on public social media posts, it overwhelmingly features cisgender, heterosexual couple dynamics.
- Contextual Variety: The scenarios are limited to commonplace settings like hotels and restaurants, leaving out other complex social environments like workplaces or nightlife.
Future work will focus on scaling the dataset to include more diverse demographic representations, a wider variety of scenarios, and broader forms of socially grounded visual reasoning beyond relationship infidelity.
Summary updated