BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games

Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, Tim Rocktäschel

TL;DR

BALROG is a new benchmark that evaluates Large Language Models (LLMs) and Vision Language Models (VLMs) as autonomous agents across six diverse, long-horizon game environments. The study reveals a significant "knowing-doing gap"—where models understand rules but fail to apply them—and shows that current multimodal models often perform worse when visual information is added to textual descriptions.

What Problem Does This Paper Address?

While LLMs have shown impressive reasoning in static, text-based tasks, their performance as "agents"—entities that must perceive, plan, and act in dynamic environments—remains a major hurdle. Most existing agentic benchmarks are limited to short-horizon tasks (like simple web navigation or office chores) involving only a few dozen interactions.

Real-world agency requires much more: spatial reasoning, long-term planning over thousands of steps, and the ability to learn environment dynamics through exploration. To bridge this gap, the authors introduced BALROG (Benchmarking Agentic LLM and VLM Reasoning On Games), a suite designed to push models to their breaking points using complex reinforcement learning environments.

Methods

The BALROG framework provides a standardized API that decouples the "Agentic Strategy" (how the model is prompted) from the underlying model itself. This allows researchers to easily swap models or test new inference-time techniques like Chain-of-Thought or few-shot learning.

Overview of the BALROG framework, highlighting the decoupling of environment wrappers from LLM clients.

Key methodological features include:

  • Dual Modalities: Models are tested in "Language-Only" (textual descriptions of the world) and "Vision-Language" (text + images) formats.
  • Fine-Grained Metrics: Instead of simple pass/fail, BALROG uses a progression-based score (0–100). For the notoriously difficult game NetHack, they developed a novel "data-informed progression system" based on the probability of a human player winning after reaching specific milestones.
  • Standardized Interaction: An env_wrapper.py standardizes observations across different game engines, while AgentQuest provides feedback to models when they attempt invalid actions.

Datasets

BALROG aggregates six established reinforcement learning environments, ranging from "seconds to master" to "years to master":

  1. BabyAI: Simple grid-world navigation and object manipulation.
  2. Crafter: A Minecraft-inspired survival and crafting game.
  3. TextWorld: Purely text-based adventure games focusing on logic and memory.
  4. Baba Is AI: A puzzle game where the player can change the rules of the world by moving word blocks.
  5. MiniHack: A sandbox for testing specific skills (navigation, combat) based on the NetHack engine.
  6. NetHack (NLE): One of the most complex "roguelike" games ever created, requiring extreme long-term planning and deep domain knowledge.

Key Insights and Ideas

  • The Vision Deficit: One of the most surprising findings was that multimodal models (VLMs) often performed worse when provided with images of the environment alongside text. This suggests that current VLM architectures struggle to integrate visual spatial data into actionable decisions, often becoming "confused" by the extra input.
  • The Knowing-Doing Gap: The authors identified a disconnect between a model's internal knowledge and its behavior. For example, GPT-4o could correctly state that eating rotten food in NetHack is fatal in a separate Q&A, yet it would frequently make the agent eat rotten food during actual gameplay.
  • Long-Horizon Failure: Models generally fail at tasks requiring irreversible logic steps, such as the Boxoban puzzles in MiniHack. They lack the "look-ahead" capability to realize a move will lead to a dead-end 50 steps later.
  • Open-Source Competitiveness: Llama 3.1 (70B) and 3.2 (90B) showed impressive performance, even outperforming GPT-4o on certain language-only tasks like Baba Is AI.

Key Results

The benchmark provides a sobering look at the current state of agentic AI. While models show promise on simpler tasks, the "boss level" (NetHack) remains largely unsolved.

  • Top Performers: Claude 3.5 Sonnet emerged as the overall leader with an average progression of ~32.6%, followed closely by GPT-4o.
  • NetHack Results: Even the most advanced models barely scratched the surface. OpenAI's o1-preview achieved the highest score at 1.57% progression—nearly triple its closest competitor, but still far from human-level play.
  • Multimodal Drop: In the Crafter environment, Llama 3.2 90B's performance dropped sharply when images were added, falling behind its smaller 11B counterpart.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Context Window Limits: A single game of NetHack can last hundreds of thousands of turns, far exceeding the context window of any current model.
  • Computational Cost: Running thousands of steps of inference for large models is expensive and slow, limiting the scale of evaluation.
  • Future Directions: The team plans to incorporate video observations and explore multi-agent collaboration. They also suggest that better "search" and "planning" algorithms at inference time (similar to the o1 series) will be necessary to conquer the most difficult environments in BALROG.

Summary updated

Abstract

Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities, however, they still struggle to perform well in complex, dynamic environments. Real-world tasks require handling intricate interactions, advanced spatial reasoning, long-term planning, and continuous exploration of new strategies—areas in which we lack effective methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, a novel benchmark designed to assess the agentic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environments with varying levels of difficulty, including tasks that are solvable by non-expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). We devise fine-grained metrics to measure performance and conduct an extensive evaluation of several popular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficiencies in vision-based decision-making, as several models perform worse when visual representations of the environments are provided. We release BALROG as an open and user-friendly benchmark to facilitate future research and development in the agentic community. Code and Leaderboard at balrogai.com
BibTeX
@inproceedings{paglieri2025balrog,
  author = {Davide Paglieri and Bartłomiej Cupiał and Samuel Coward and Ulyana Piterbarg and Maciej Wolczyk and Akbir Khan and Eduardo Pignatelli and Łukasz Kuciński and Lerrel Pinto and Rob Fergus and Jakob Nicolaus Foerster and Jack Parker-Holder and Tim Rocktäschel},
  title = {{BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=fp6t3F669F},
}
Figure