TL;DR
BALROG is a new benchmark that evaluates Large Language Models (LLMs) and Vision Language Models (VLMs) as autonomous agents across six diverse, long-horizon game environments. The study reveals a significant "knowing-doing gap"—where models understand rules but fail to apply them—and shows that current multimodal models often perform worse when visual information is added to textual descriptions.
What Problem Does This Paper Address?
While LLMs have shown impressive reasoning in static, text-based tasks, their performance as "agents"—entities that must perceive, plan, and act in dynamic environments—remains a major hurdle. Most existing agentic benchmarks are limited to short-horizon tasks (like simple web navigation or office chores) involving only a few dozen interactions.
Real-world agency requires much more: spatial reasoning, long-term planning over thousands of steps, and the ability to learn environment dynamics through exploration. To bridge this gap, the authors introduced BALROG (Benchmarking Agentic LLM and VLM Reasoning On Games), a suite designed to push models to their breaking points using complex reinforcement learning environments.
Methods
The BALROG framework provides a standardized API that decouples the "Agentic Strategy" (how the model is prompted) from the underlying model itself. This allows researchers to easily swap models or test new inference-time techniques like Chain-of-Thought or few-shot learning.
Key methodological features include:
- Dual Modalities: Models are tested in "Language-Only" (textual descriptions of the world) and "Vision-Language" (text + images) formats.
- Fine-Grained Metrics: Instead of simple pass/fail, BALROG uses a progression-based score (0–100). For the notoriously difficult game NetHack, they developed a novel "data-informed progression system" based on the probability of a human player winning after reaching specific milestones.
- Standardized Interaction: An
env_wrapper.pystandardizes observations across different game engines, whileAgentQuestprovides feedback to models when they attempt invalid actions.
Datasets
BALROG aggregates six established reinforcement learning environments, ranging from "seconds to master" to "years to master":
- BabyAI: Simple grid-world navigation and object manipulation.
- Crafter: A Minecraft-inspired survival and crafting game.
- TextWorld: Purely text-based adventure games focusing on logic and memory.
- Baba Is AI: A puzzle game where the player can change the rules of the world by moving word blocks.
- MiniHack: A sandbox for testing specific skills (navigation, combat) based on the NetHack engine.
- NetHack (NLE): One of the most complex "roguelike" games ever created, requiring extreme long-term planning and deep domain knowledge.
Key Insights and Ideas
- The Vision Deficit: One of the most surprising findings was that multimodal models (VLMs) often performed worse when provided with images of the environment alongside text. This suggests that current VLM architectures struggle to integrate visual spatial data into actionable decisions, often becoming "confused" by the extra input.
- The Knowing-Doing Gap: The authors identified a disconnect between a model's internal knowledge and its behavior. For example, GPT-4o could correctly state that eating rotten food in NetHack is fatal in a separate Q&A, yet it would frequently make the agent eat rotten food during actual gameplay.
- Long-Horizon Failure: Models generally fail at tasks requiring irreversible logic steps, such as the Boxoban puzzles in MiniHack. They lack the "look-ahead" capability to realize a move will lead to a dead-end 50 steps later.
- Open-Source Competitiveness: Llama 3.1 (70B) and 3.2 (90B) showed impressive performance, even outperforming GPT-4o on certain language-only tasks like Baba Is AI.
Key Results
The benchmark provides a sobering look at the current state of agentic AI. While models show promise on simpler tasks, the "boss level" (NetHack) remains largely unsolved.
- Top Performers: Claude 3.5 Sonnet emerged as the overall leader with an average progression of ~32.6%, followed closely by GPT-4o.
- NetHack Results: Even the most advanced models barely scratched the surface. OpenAI's o1-preview achieved the highest score at 1.57% progression—nearly triple its closest competitor, but still far from human-level play.
- Multimodal Drop: In the Crafter environment, Llama 3.2 90B's performance dropped sharply when images were added, falling behind its smaller 11B counterpart.
Limitations and Future Work
The authors acknowledge several hurdles:
- Context Window Limits: A single game of NetHack can last hundreds of thousands of turns, far exceeding the context window of any current model.
- Computational Cost: Running thousands of steps of inference for large models is expensive and slow, limiting the scale of evaluation.
- Future Directions: The team plans to incorporate video observations and explore multi-agent collaboration. They also suggest that better "search" and "planning" algorithms at inference time (similar to the o1 series) will be necessary to conquer the most difficult environments in BALROG.
Summary updated