CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning

Ming Li, Chenguang Wang, Yijun Liang, Xiyao Wang, Yuhang Zhou, Xiyang Wu, Yuqing Zhang, Ruiyi Zhang, Tianyi Zhou

TL;DR

While state-of-the-art Multi-Modal Large Language Models (MLLMs) like GPT-o3 can easily ace complex visual tasks like GeoGuessr, their performance plummets to near-zero when tasked with finding subtle, context-dependent anomalies. This paper introduces CaughtCheating, a novel benchmark inspired by social media sleuths trying to identify hidden clues that contradict a partner's claim (e.g., finding a second person's reflection in a spoon when someone claims to be dining alone). The findings reveal a critical gap in MLLMs' "detective-level" visual perception and active exploration capabilities.

What Problem Does This Paper Address?

Recent frontier MLLMs have achieved near-ceiling scores on standard vision-language benchmarks. They can even out-perform most humans on highly specialized tasks, such as pinpointing geographic locations from obscure visual cues or finding expertly camouflaged animals.

Figure 2: Demonstration of GPT-o3’s multimodal visual-reasoning breadth. (a) Visual search: locating Waldo in a densely populated illustration. (b) Visual search for camouflage: spotting a nearly invisible copperhead snake hidden among dry leaves. (c) GeoGuessr: identifying the upper terminal of Chair 1 at New Mexico, and estimating its latitude/longitude from a single image. (d) TimeGuesser: dating the photograph by matching architectural signage and period vehicles to museum and heritage records. These examples highlight o3’s strong visual perception and reasoning capacity across various visual tasks that most humans can not accomplish.

However, the authors identify a major blind spot: detective-level visual reasoning. Excellent human detectives excel at noticing minuscule, low-salience cues and weaving them into situational, socially grounded explanations. Current benchmarks fail to test this because they explicitly define the target object in the prompt. This paper investigates whether MLLMs can actively explore an image to find unguided, context-dependent anomalies that violate a specific social claim.

Methods

Rather than introducing a new architecture, the authors establish a rigorous evaluation framework grounded in cognitive science—specifically, Guided Search Theory. This theory states that visual attention is driven by factors like bottom-up salience (how much an item "pops out") and top-down feature guidance (knowing what features to look for).

To diagnose exactly why models fail, the authors implement a Decomposition Analysis. For every image, they generate two types of sub-questions:

  1. Decomposed Perception Questions: Test if the model can see the clue when explicitly told where to look (e.g., "Is there a pair of slippers under the table?").
  2. Decomposed Reasoning Questions: Test if the model understands the social implication of the clue (e.g., "Does the presence of these slippers contradict the claim of being alone?").

This allows researchers to isolate whether a model fails because it is visually blind to the clue, or because it simply doesn't understand its contextual significance.

Datasets

The authors introduce the CaughtCheating benchmark, consisting of 100 real-world images sourced from public social media posts. The dataset is split evenly:

  • Clued Category (50 images): Contains a definitive, deterministic clue that violates a stated claim, along with weaker non-deterministic clues.
  • Unclued Category (50 images): Innocent images containing no evidence of deception.

The images primarily feature common real-world settings: hotels (69%), dining venues (29%), and karaoke bars (2%). The authors manually screened the data and applied minimal cropping to preserve subtle indicators of another person's presence while obscuring obvious giveaways. Notably, they explicitly avoided synthetic AI-generated images because current text-to-image models make hidden clues far too overt and obvious to detect.

An example of a Clued image where a girl's reflection is subtly visible in the spoon.

Key Insights and Ideas

  • The Failure of Exhaustive Grid Search: Advanced models like GPT-o3 approach visual reasoning via a methodical, quadrant-by-quadrant grid search. While highly effective for finding Waldo, this strategy fails in real-world contexts where the target object isn't predefined. Lacking top-down feature guidance, the model simply doesn't know what constitutes an anomaly.
  • The "Accuse Everything" Bias: When evaluating innocent images (the Unclued category), advanced reasoning models suffer from severe overthinking and hallucination. Instead of concluding that there is no evidence, they tend to flag benign details—like the amount of food or the number of cups—as highly suspicious.
  • Perception is Not the Bottleneck: The decomposition analysis revealed that models achieve high accuracy on Perception questions when directed to the clue. The true bottleneck is the active exploration strategy—knowing where to look and how to connect a localized detail back to the overarching social claim.
  • Safety Alignment Interference: Interestingly, proprietary models like GPT-4o and GPT-o3 saw degraded reasoning scores because their safety filters caused them to refuse to answer gender-related questions, which are often central to relationship-based context clues.

Key Results

The benchmark proved to be incredibly challenging for even the most advanced models:

  • GPT-o3, the top-performing model, achieved an overall F1 score of just 23.9%.
  • On the Clued category, GPT-o3 managed 26.0% accuracy in detecting deterministic clues, with an Intersection over Union (IoU) of 17.2%.
  • On the Unclued category, GPT-o3’s accuracy dropped to a dismal 8.0%, driven by its tendency to hallucinate suspicion in innocent photos.
  • In comparison, human participants achieved a 59.6% overall F1 score (56.0% Clued Accuracy and 68.0% Unclued Accuracy), vastly outperforming all agentic MLLMs.
Case studies showing how models perform across difficult clued, easy clued, and unclued images.

Limitations and Future Work

The authors acknowledge several limitations tied to data scarcity:

  1. Sample Size: The dataset is relatively small (100 images) due to the difficulty of sourcing and verifying real-world examples.
  2. Demographic Bias: Because the dataset relies on public social media posts, it overwhelmingly features cisgender, heterosexual couple dynamics.
  3. Contextual Variety: The scenarios are limited to commonplace settings like hotels and restaurants, leaving out other complex social environments like workplaces or nightlife.

Future work will focus on scaling the dataset to include more diverse demographic representations, a wider variety of scenarios, and broader forms of socially grounded visual reasoning beyond relationship infidelity.

Summary updated

Abstract

Recent agentic Multi-Modal Large Language Models (MLLMs) such as GPT-o3 have achieved near-ceiling scores on various existing benchmarks, motivating a demand for more challenging test tasks. These MLLMs have been reported to excel in a few expert-level tasks for humans, e.g., GeoGuesser, reflecting their potential as a detective who can notice minuscule cues in an image and weave them into coherent, situational explanations, leading to a reliable answer. But can they match the performance of excellent human detectives? To answer this question, we investigate some hard scenarios where GPT-o3 can still handle, and find a common scenario where o3's performance drops to nearly zero, which we name CaughtCheating. It is inspired by the social media requests that ask others to detect suspicious clues from photos shared by the poster's partner. We conduct extensive experiments and analysis to understand why existing MLLMs lack sufficient capability to solve this kind of task. CaughtCheating provides a class of challenging visual perception and reasoning tasks with great value and practical usage. Success in these tasks paves the way for MLLMs to acquire human-level detective perception and reasoning capabilities.
BibTeX
@inproceedings{li2025caughtcheating,
  author = {Ming Li and Chenguang Wang and Yijun Liang and Xiyao Wang and Yuhang Zhou and Xiyang Wu and Yuqing Zhang and Ruiyi Zhang and Tianyi Zhou},
  title = {{CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning}},
  booktitle = {LAW @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=2CrQvJBXq9},
}
Figure