TL;DR
While frontier multimodal reasoning models like GPT-o3 and Gemini-2.5 Pro excel at complex visual tasks like GeoGuessr, they crumble when faced with open-ended detective tasks. The new CaughtCheating benchmark tests an AI's ability to spot subtle, context-dependent clues in photos that contradict a partner's claims (e.g., a faint reflection in a spoon debunking a claim of "dining alone"). Even the strongest models achieve an F1 score of only 23.9% on real-world cases, either overlooking subtle visual evidence or hallucinating infidelity in innocent images.
What Problem Does This Paper Address?
Recent multimodal large language models (MLLMs) demonstrate impressive reasoning when tasks provide clear targets—such as finding Waldo, locating camouflaged animals, or geolocating an obscure landmark. These tasks typically allow models to rely on exhaustive grid searches paired with explicit top-down visual goals (e.g., looking for red-and-white stripes).
However, real-world detective work requires finding evidence when the target is not predefined and its significance is entirely context-dependent. Inspired by social media posts where users ask others to inspect photos for signs of cheating, the authors ask: Do MLLMs possess genuine detective-level perception and social reasoning, or do they merely brute-force feature-matching?
Methods
To understand why models struggle, the authors ground their evaluation in Guided Search Theory from cognitive science (Wolfe et al.), which identifies key drivers of visual attention:
- Low Bottom-Up Salience: Clues (tiny reflections, stray personal items, partially cropped limbs) blend into the background.
- Lack of Top-Down Guidance: The model is not told what to look for—an anomaly could be anything from two wine glasses to an out-of-place phone charger.
- Complex Scene Meaning: Identifying a clue requires interpreting subtle social norms and contradictions against a specific claim.
Evaluation Framework
- Primary Inquiry: Models are given an image and a partner's claim (e.g., "My boyfriend said he's dining alone...") and asked if anything is suspicious.
- Decomposed Queries: To separate failure of perception from failure of reasoning, images are annotated with targeted sub-questions:
- Perception Questions: Can the model detect the clue when explicitly told what and where it is?
- Reasoning Questions: Does the model understand why that clue implies a contradiction?
- Automated Judge: An evaluation pipeline powered by GPT-4.1 scores open-ended responses against ground-truth deterministic and non-deterministic clues (demonstrating strong human agreement with Cohen’s $\kappa = 0.82$ to $0.94$).
Datasets
The authors introduce the CaughtCheating benchmark, split into two complementary subsets:
- Real Subset (100 images): Sourced from public social media posts across hotel, dining, and karaoke scenes. Split evenly into Clued (50 images containing subtle, verifiable contradictions) and Unclued (50 innocent baseline images). Minimal cropping was applied to some photos to obscure obvious cues, keeping them challenging for AI while human-interpretable.
- Synthetic Subset (3,700 images): Generated using GPT-Image-1 across 10 scenes (e.g., gyms, offices, bars, cars) and balanced by photographer gender, containing 1,850 clued and 1,850 unclued samples.
Key Insights and Ideas
- Grid Search Fails Without Semantic Targets: When analyzing a food photo, GPT-o3 methodically zooms in on pizza slices, plates, and cutlery. Yet even when its attention crops directly onto a spoon, it fails to perceive or interpret the reflection of another diner seated across the table.
- Models Can See, But Cannot Discover: On decomposed perception questions (where the clue is pointed out explicitly), GPT-o3 scores 62.0% and Gemini-2.5 Pro scores 80.0%. However, without hints, their ability to independently detect the clue plummets to 20–26%.
- The "Paranoid Detective" Phenomenon: Advanced reasoning models severely over-accuse. On innocent ("Unclued") photos, GPT-o3 achieves an accuracy of just 8.0% (Gemini-2.5 Pro: 22.0%), hallucinating suspicious meanings behind ordinary table settings, portion sizes, or standard hotel amenities.
Key Results
| Model | Clued Acc (%) ↑ | Clued IoU (%) ↑ | Unclued Acc (%) ↑ | Real Overall F1 (%) ↑ | Synthetic F1 (%) ↑ |
|---|---|---|---|---|---|
| InternVL2.5-72B | 2.0 | 0.8 | 80.0 | 3.3 | 56.2 |
| GPT-4o | 4.0 | 1.0 | 54.0 | 5.3 | 64.2 |
| Gemini-2.5-pro | 20.0 | 15.1 | 22.0 | 20.2 | 79.2 |
| GPT-o3 | 26.0 | 17.2 | 8.0 | 23.9 | 79.6 |
| Human Baseline | 56.0 | — | 68.0 | 59.6 | 96.4 |
- Real Subset: Even the top-performing model, GPT-o3, achieves only a 23.9% F1 score (compared to 59.6% for humans). Most open-source and non-reasoning models scored near zero on Clued Accuracy.
- Synthetic vs. Real Gap: SOTA models reach ~80% F1 on the synthetic subset. Current image generators produce cleaner images where clues are unnaturally prominent, failing to replicate the organic, optical subtlety of real camera captures.
Limitations and Future Work
- Data Sourcing and Demographics: Sourcing exclusively from public social media biased the dataset heavily toward cisgender, heterosexual relationships and common settings (hotels and restaurants), leaving out diverse relationship structures and niche environments.
- Synthetic Generation Gaps: Current image synthesis models cannot faithfully generate optical anomalies (e.g., complex curvature distortions or faint glass reflections), limiting synthetic data's utility as a true proxy for hard real-world cases.
- Narrow Social Context: The benchmark investigates a specific, socially charged scenario (infidelity). Future work should expand these unconstrained, open-ended detective evaluations to broader safety, forensic, and visual commonsense domains.
Summary updated