VisEscape: A Benchmark for Evaluating Exploration-driven Decision-making in Virtual Escape Rooms

Seungwon Lim, Sungwoong Kim, Jihwan Yu, Sungjae Lee, Jiwan Chung, Youngjae Yu

TL;DR

This paper introduces VisEscape, a novel benchmark of 20 virtual escape rooms designed to rigorously test multimodal AI agents' ability to explore, reason, and make decisions in dynamic environments. The authors found that even state-of-the-art models struggle with this task, highlighting the critical need for improved memory management and reasoning capabilities for agents to succeed in complex, exploration-driven scenarios.

What Problem Does This Paper Address?

Realistic decision-making often requires agents to actively explore their environment, gather information, and infer goals and solutions, rather than relying on predefined instructions. While previous research has explored embodied agents in tasks with explicit goals, there's a significant gap in evaluating their ability to conduct self-directed exploration and iterative hypothesis formulation in dynamic settings. This paper addresses this gap by proposing VisEscape, a benchmark that simulates the cognitive challenges of real-world escape rooms, demanding agents to build spatial-temporal knowledge, infer clues, and adapt to changing conditions to achieve an implicit goal: "escape the room."

Methods

The core of the VisEscape benchmark involves creating 20 distinct virtual escape rooms. The construction process includes:

  • Object Generation: Creating 3D models for receptacles (e.g., drawers, safes) and items (e.g., keys, clues).
  • Game Engine Design: Building a game graph using a Finite State Machine (FSM) to track room states and transitions based on agent actions.
  • Scene Generation: Arranging objects within rooms and mapping visual observations to game states.
  • Visual Quizzes: Integrating one to two number or letter code locks paired with corresponding visual quizzes to test associative thinking and inference.

To evaluate AI agents, the authors augmented existing models with:

  • Memory Management Module: This module comprises structured spatial memory, exploration memory, and salient action memory to help agents build a coherent mental map, avoid redundant actions, and remember past attempts.
  • Reasoning Module: This module facilitates pre-action reasoning (hypothesis formulation) and post-action reasoning (hypothesis verification), enabling agents to analyze outcomes and adapt their strategies.

Datasets

The paper introduces VisEscape, a new benchmark consisting of 20 virtual escape rooms. Each room is designed to be visually rich and dynamically changing, requiring agents to explore, interact with objects, and solve puzzles. The dataset includes:

  • 32 different receptacles.
  • 23 different items.
  • One or two visual quizzes per room.

The authors also conducted a human study on a subset of these rooms to establish a baseline for comparison.

Key Insights and Ideas

  • Exploration is Key: Success in VisEscape hinges on an agent's ability to actively explore and build a dynamic understanding of the environment, not just solve isolated puzzles.
  • Memory and Reasoning Synergy: Integrating memory management and reasoning modules significantly boosts agent performance, demonstrating a synergistic effect where memory supports reasoning and vice-versa.
  • Current MLLMs Fall Short: Even advanced multimodal large language models (MLLMs) struggle significantly with VisEscape, exhibiting low escape rates and varied problem-solving strategies, highlighting a critical area for improvement.
  • Brute-Force is Inefficient: Smaller models often resort to brute-force strategies for visual quizzes, which is less effective than grounded reasoning and leads to lower efficiency and success rates.
  • Iterative Hypothesis Testing is Crucial: The ability to repeatedly formulate and test hypotheses, informed by past actions and observations, is vital for overcoming challenges and achieving success.

Key Results

  • Low Baseline Performance: State-of-the-art models like Claude-3.5-Sonnet and GPT-40 achieved success rates below 10% without any hints. Open-source models performed even worse, with most failing to achieve any escapes.
  • Significant Improvement with Modules: Applying memory management and reasoning modules led to substantial improvements, with agents achieving x2.4 enhanced effectiveness and x3.8 improved efficiency.
  • Human Performance: Humans achieved a high success rate (82.5%) and goal completion (95.8%) with an average of 52.8 steps, serving as a strong benchmark.
  • Reasoning Boosts Visual Quizzes: The reasoning module significantly improved visual quiz accuracy, especially for frontier models.
  • Efficiency Gains: Models equipped with memory and reasoning not only achieved more but also did so more efficiently, reducing trajectory length and improving SPL.

Limitations and Future Work

  • Generalizability: The specific task of room escapes might not fully generalize to all real-world environments, which can be more complex and unfamiliar.
  • Discrete Action Space: VisEscape uses a discrete action space, and converting these findings to continuous action sequences for real-world execution remains a challenge.
  • Future Directions: The authors suggest further research into developing more capable AI systems that integrate memory and adaptive reasoning for complex, dynamic tasks. They also point to the potential of LLMs with strong reasoning capabilities combined with VLMs for enhanced multimodal reasoning.

Summary updated

Abstract

Escape rooms present a unique cognitive challenge that demands exploration-driven planning: with the sole instruction to 'escape the room', players must actively search their environment, collecting information, and finding solutions through repeated trial and error. Motivated by this, we introduce VisEscape, a benchmark of 20 virtual escape rooms specifically designed to evaluate AI models under these challenging conditions, where success depends not only on solving isolated puzzles but also on iteratively constructing and refining spatial-temporal knowledge of a dynamically changing environment. On VisEscape, we observe that even state-of-the-art multi-modal models generally fail to escape the rooms, showing considerable variation in their progress and problem-solving approaches. We find that integrating memory management and reasoning contributes to efficient exploration and enables successive hypothesis formulation and testing, thereby leading to significant improvements in dynamic and exploration-driven environments
BibTeX
@inproceedings{lim2025visescape,
  author = {Seungwon Lim and Sungwoong Kim and Jihwan Yu and Sungjae Lee and Jiwan Chung and Youngjae Yu},
  title = {{VisEscape: A Benchmark for Evaluating Exploration-driven Decision-making in Virtual Escape Rooms}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure