Abstract
Escape rooms present a unique cognitive challenge that demands exploration-driven planning: players should actively search their environment, continuously update their knowledge based on new discoveries, and connect disparate clues to determine which elements are relevant to their objectives. Motivated by this, we introduce VisEscape, a benchmark of 20 virtual escape rooms specifically designed to evaluate AI models under these challenging conditions, where success depends not only on solving isolated puzzles but also on iteratively constructing and refining spatial-temporal knowledge of a dynamically changing environment. On VisEscape, we observe that even state-of-the-art multimodal models generally fail to escape the rooms, showing considerable variation in their levels of progress and trajectories. To address this issue, we propose VisEscaper, which effectively integrates Memory, Feedback, and ReAct modules, demonstrating significant improvements by performing 3.7 times more effectively and 4.9 times more efficiently on average compared to baseline agents.
1. Introduction: The Challenge of Exploration
Most current AI benchmarks for embodied agents focus on "explicit instruction following," where goals and steps are largely predefined (e.g., "put the apple in the microwave"). However, real-world intelligence often requires exploration-driven decision-making. In these scenarios, an agent must autonomously map its environment, gather clues, and refine plans through trial and error.
Escape rooms serve as an ideal proxy for this cognitive challenge. To succeed, an agent must connect disparate pieces of information—such as matching a color code found on a wall to a locked box on a desk—while maintaining a consistent mental model of a dynamically changing space.
2. The VisEscape Benchmark
The VisEscape benchmark consists of 20 unique virtual rooms built using 3D modeling and photorealistic rendering. Each room is designed with complex logic, containing an average of 8.4 "checkpoints" or bottleneck tasks that must be completed in sequence. A perfect "oracle" solution typically requires approximately 27 steps.
Central to the benchmark are "visual quizzes"—hand-crafted puzzles requiring associative thinking. These quizzes force the agent to interpret visual data (like clock faces or geometric patterns) and apply that knowledge to interact with objects elsewhere in the room.
3. The VisEscaper Framework
To address the limitations of current Multimodal Large Language Models (MLLMs), we propose VisEscaper, which augments agents with two primary components:
- Memory Management Module: Maintains structured spatial memory (a map of the four walls), exploration memory (tracking inspected areas), and salient action memory (recording critical interactions like passcode attempts).
- Reasoning Module: Implements a "think-before-you-act" (hypothesis generation) and "reflect-after-you-act" (outcome evaluation) loop.
We also explored a Socratic approach, where visual scenes are converted into text captions for processing by high-reasoning LLMs (e.g., DeepSeek-R1), which often yields superior logical consistency compared to direct visual processing by VLMs.
4. Results and Analysis
Baseline performance for state-of-the-art MLLMs was notably low. Claude 3.5 Sonnet and GPT-4o achieved success rates (SR) of less than 10%, while most open-source models failed to complete any rooms (0% SR). Common failure modes included the "brute-force trap," where agents without memory fell into repetitive loops.
The integration of VisEscaper modules led to dramatic improvements. For instance, Gemini 1.5 Pro improved from 0% to a 25% success rate, and Claude 3.5 Sonnet reached 21.7%. Overall, VisEscaper performed 3.7 times more effectively and 4.9 times more efficiently than baseline agents. Despite these gains, a significant gap remains compared to the human baseline of 82.5% SR, particularly in solving complex visual quizzes that require cross-room association.
5. Conclusion
VisEscape highlights that while AI models are proficient at following explicit orders, they struggle with self-directed exploration and iterative hypothesis testing. By providing a structured benchmark and a framework for integrated memory and reasoning, we aim to push the development of agents capable of true autonomous problem-solving in complex, unknown environments.
Summary updated