CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning

Chenguang Wang, Ming Li, Yijun Liang, Xiyao Wang, Yuhang Zhou, Xiyang Wu, Yuqing Zhang, Ruiyi Zhang, Tianyi ZhouSubmission to ICLR 2026 · Withdrawn

TL;DR

While frontier multimodal reasoning models like GPT-o3 and Gemini-2.5 Pro excel at complex visual tasks like GeoGuessr, they crumble when faced with open-ended detective tasks. The new CaughtCheating benchmark tests an AI's ability to spot subtle, context-dependent clues in photos that contradict a partner's claims (e.g., a faint reflection in a spoon debunking a claim of "dining alone"). Even the strongest models achieve an F1 score of only 23.9% on real-world cases, either overlooking subtle visual evidence or hallucinating infidelity in innocent images.

What Problem Does This Paper Address?

Recent multimodal large language models (MLLMs) demonstrate impressive reasoning when tasks provide clear targets—such as finding Waldo, locating camouflaged animals, or geolocating an obscure landmark. These tasks typically allow models to rely on exhaustive grid searches paired with explicit top-down visual goals (e.g., looking for red-and-white stripes).

However, real-world detective work requires finding evidence when the target is not predefined and its significance is entirely context-dependent. Inspired by social media posts where users ask others to inspect photos for signs of cheating, the authors ask: Do MLLMs possess genuine detective-level perception and social reasoning, or do they merely brute-force feature-matching?

Methods

To understand why models struggle, the authors ground their evaluation in Guided Search Theory from cognitive science (Wolfe et al.), which identifies key drivers of visual attention:

  1. Low Bottom-Up Salience: Clues (tiny reflections, stray personal items, partially cropped limbs) blend into the background.
  2. Lack of Top-Down Guidance: The model is not told what to look for—an anomaly could be anything from two wine glasses to an out-of-place phone charger.
  3. Complex Scene Meaning: Identifying a clue requires interpreting subtle social norms and contradictions against a specific claim.

Evaluation Framework

  • Primary Inquiry: Models are given an image and a partner's claim (e.g., "My boyfriend said he's dining alone...") and asked if anything is suspicious.
  • Decomposed Queries: To separate failure of perception from failure of reasoning, images are annotated with targeted sub-questions:
  • Perception Questions: Can the model detect the clue when explicitly told what and where it is?
  • Reasoning Questions: Does the model understand why that clue implies a contradiction?
  • Automated Judge: An evaluation pipeline powered by GPT-4.1 scores open-ended responses against ground-truth deterministic and non-deterministic clues (demonstrating strong human agreement with Cohen’s $\kappa = 0.82$ to $0.94$).

Datasets

The authors introduce the CaughtCheating benchmark, split into two complementary subsets:

  • Real Subset (100 images): Sourced from public social media posts across hotel, dining, and karaoke scenes. Split evenly into Clued (50 images containing subtle, verifiable contradictions) and Unclued (50 innocent baseline images). Minimal cropping was applied to some photos to obscure obvious cues, keeping them challenging for AI while human-interpretable.
  • Synthetic Subset (3,700 images): Generated using GPT-Image-1 across 10 scenes (e.g., gyms, offices, bars, cars) and balanced by photographer gender, containing 1,850 clued and 1,850 unclued samples.

Key Insights and Ideas

  • Grid Search Fails Without Semantic Targets: When analyzing a food photo, GPT-o3 methodically zooms in on pizza slices, plates, and cutlery. Yet even when its attention crops directly onto a spoon, it fails to perceive or interpret the reflection of another diner seated across the table.
  • Models Can See, But Cannot Discover: On decomposed perception questions (where the clue is pointed out explicitly), GPT-o3 scores 62.0% and Gemini-2.5 Pro scores 80.0%. However, without hints, their ability to independently detect the clue plummets to 20–26%.
  • The "Paranoid Detective" Phenomenon: Advanced reasoning models severely over-accuse. On innocent ("Unclued") photos, GPT-o3 achieves an accuracy of just 8.0% (Gemini-2.5 Pro: 22.0%), hallucinating suspicious meanings behind ordinary table settings, portion sizes, or standard hotel amenities.

Key Results

Model Clued Acc (%) ↑ Clued IoU (%) ↑ Unclued Acc (%) ↑ Real Overall F1 (%) ↑ Synthetic F1 (%) ↑
InternVL2.5-72B 2.0 0.8 80.0 3.3 56.2
GPT-4o 4.0 1.0 54.0 5.3 64.2
Gemini-2.5-pro 20.0 15.1 22.0 20.2 79.2
GPT-o3 26.0 17.2 8.0 23.9 79.6
Human Baseline 56.0 — 68.0 59.6 96.4
  • Real Subset: Even the top-performing model, GPT-o3, achieves only a 23.9% F1 score (compared to 59.6% for humans). Most open-source and non-reasoning models scored near zero on Clued Accuracy.
  • Synthetic vs. Real Gap: SOTA models reach ~80% F1 on the synthetic subset. Current image generators produce cleaner images where clues are unnaturally prominent, failing to replicate the organic, optical subtlety of real camera captures.

Limitations and Future Work

  • Data Sourcing and Demographics: Sourcing exclusively from public social media biased the dataset heavily toward cisgender, heterosexual relationships and common settings (hotels and restaurants), leaving out diverse relationship structures and niche environments.
  • Synthetic Generation Gaps: Current image synthesis models cannot faithfully generate optical anomalies (e.g., complex curvature distortions or faint glass reflections), limiting synthetic data's utility as a true proxy for hard real-world cases.
  • Narrow Social Context: The benchmark investigates a specific, socially charged scenario (infidelity). Future work should expand these unconstrained, open-ended detective evaluations to broader safety, forensic, and visual commonsense domains.

Summary updated

Abstract

Recent agentic Multi-Modal Large Language Models (MLLMs) such as GPT-o3 have achieved near-ceiling scores on various existing benchmarks, motivating a demand for more challenging test tasks. These MLLMs have been reported to excel in a few expert-level tasks for humans, e.g., GeoGuesser, reflecting their potential as a detective who can notice minuscule cues in an image and weave them into coherent, situational explanations, leading to a reliable answer. But can they match the performance of excellent human detectives? To answer this question, we investigate some hard scenarios where GPT-o3 can still handle, and find a common scenario where o3's performance drops to nearly zero, which we name CaughtCheating. It is inspired by the social media requests that ask others to detect suspicious clues from photos shared by the poster's partner. We conduct extensive experiments and analysis to understand why existing MLLMs lack sufficient capability to solve this kind of task. CaughtCheating provides a class of challenging visual perception and reasoning tasks with great value and practical usage. Success in these tasks paves the way for MLLMs to acquire human-level detective perception and reasoning capabilities.
BibTeX
@inproceedings{wang2025caughtcheating,
  author = {Chenguang Wang and Ming Li and Yijun Liang and Xiyao Wang and Yuhang Zhou and Xiyang Wu and Yuqing Zhang and Ruiyi Zhang and Tianyi Zhou},
  title = {{CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning}},
  booktitle = {LAW @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=2CrQvJBXq9},
}
Figure