Dyna-Mind: Learning to Simulate from Experience for Better AI Agents

Xiao Yu, Baolin Peng, Michel Galley, Hao Cheng, Qianhui Wu, Janardhan Kulkarni, Suman Nath, Zhou Yu, Jianfeng Gao

TL;DR

Dyna-Mind is a two-stage training framework that equips AI agents with "vicarious trial and error"—the ability to mentally simulate potential futures before taking an action. By grounding reasoning in real environment interactions (ReSim) and refining those simulations through a novel reinforcement learning objective (Dyna-GRPO), the authors significantly boost agent performance in complex, long-horizon tasks like Android navigation.

The Missing Piece in AI Reasoning: World Modeling

While Large Language Models (LLMs) have reached expert-level proficiency in static domains like mathematics and coding, they often stumble in interactive, "long-horizon" environments. Whether navigating a smartphone to book a flight or solving a complex spatial puzzle, current agents tend to be reactive. They look at the current screen and pick an action without truly understanding the consequences.

Even advanced "reasoning" models like DeepSeek-R1, which utilize Chain-of-Thought (CoT), often struggle to accurately predict how an environment will change after an action. This "world modeling" gap leads to cascading errors. The authors argue that for an agent to be truly effective, it needs the cognitive ability to perform "vicarious trial and error"—mentally simulating alternative futures to pick the best path.

The Dyna-Mind Framework

To bridge this gap, the authors introduce Dyna-Mind, a framework divided into two distinct training stages designed to explicitly teach agents how to simulate and plan.

Overview of the Dyna-Mind framework featuring Stage 1 (ReSim) and Stage 2 (Dyna-GRPO)

Stage 1: Reasoning with Simulations (ReSim)

The goal of the first stage is to ground the agent's internal "thought process" in actual world dynamics. Instead of just asking a model to "think" in a vacuum, ReSim uses a search algorithm (like Depth-First Search) to explore a real environment.

ReSim data collection process showing how search trees are converted into structured reasoning traces

The system collects these real interaction trees and uses a high-level Vision-Language Model (VLM) to aggregate them into a single, structured reasoning trace. The agent is then fine-tuned on these traces, learning to generate "mental simulations" that are faithful to how the world actually works.

Stage 2: Dyna-GRPO

The second stage further refines the policy and simulation accuracy using online Reinforcement Learning. Building on Group Relative Policy Optimization (GRPO), Dyna-Mind introduces a "Simulation Improvement" loop where the model performs SimRollouts. In these rollouts, the model is temporarily given access to ground-truth future states to help it refine its reasoning. As a result, the model learns to better integrate its simulation ability into its decision-making, rewarded not just for the final outcome, but for how well its internal plan aligns with reality.

Key Insights and Performance Gains

The framework was evaluated on three distinct benchmarks: Sokoban (spatial puzzles), ALFWorld (text-based household tasks), and AndroidWorld (realistic smartphone OS navigation).

  • Massive Success Rate Improvements: On ALFWorld, Dyna-Mind reached a 90.8% success rate, vastly outperforming the REACT-based DeepSeek-R1 (62.5%). On the highly realistic AndroidWorld benchmark, Dyna-Mind achieved a 31.8% average success rate, nearly doubling the performance of the Stage 1 model and significantly outperforming GPT-4o (5.1%) and Qwen2.5-VL baselines.
  • Efficiency Through Accuracy: Interestingly, the ReSim-trained models achieved better performance than DeepSeek-R1 while using 11x fewer tokens for their reasoning traces. This suggests that "thinking" doesn't need to be excessively long; it just needs to be accurate and grounded.
  • Real Experience Beats Synthetic Distillation: Previous methods tried to distill reasoning from "teacher" models. Dyna-Mind shows that training on traces built from actual environment feedback creates more reliable agents.
  • Improved Simulation Accuracy: The "Sim Score" (measuring how well the agent's predicted next state matched the actual one) improved significantly after Dyna-GRPO training, confirming the RL stage successfully taught the model to "see" the future more clearly.

Limitations and Future Directions

While Dyna-Mind represents a significant leap forward, challenges remain. Training on environments like AndroidWorld is extremely compute-intensive, with episodes taking 15-20 minutes. Furthermore, the agent's performance is still bottlenecked by the base vision-language model's ability to understand complex UI elements, and agents still struggle to recover if they make a sequence of multiple mistakes. Future work may explore more complex search algorithms (like Monte Carlo Tree Search) within the ReSim framework and focus on improving the underlying foundation models' GUI grounding.

Summary updated

Abstract

Reasoning models have recently shown remarkable progress in domains such as math and coding. However, their expert-level abilities in math and coding contrast sharply with their performance in long-horizon, interactive tasks such as web navigation and computer/phone-use. Inspired by literature on human cognition, we argue that current AI agents need ``vicarious trial and error'' - the capacity to mentally simulate alternative futures before acting - in order to enhance their understanding and performance in complex interactive environments. We introduce Dyna-Mind, a two-stage training framework that explicitly teaches (V)LM agents to integrate such simulation into their reasoning. In stage 1, we introduce Reasoning with Simulations (ReSim), which trains the agent to generate structured reasoning traces from expanded search trees built from real experience gathered through environment interactions. ReSim thus grounds the agent's reasoning in faithful world dynamics and equips it with the ability to anticipate future states in its reasoning. In stage 2, we propose Dyna-GRPO, an online reinforcement learning method to further strengthen the agent's simulation and decision-making ability by using both outcome rewards and intermediate states as feedback from real rollouts. Experiments on two synthetic benchmarks (Sokoban and ALFWorld) and one realistic benchmark (AndroidWorld) demonstrate that (1) ReSim effectively infuses simulation ability into AI agents, and (2) Dyna-GRPO leverages outcome and interaction-level signals to learn better policies for long-horizon, planning-intensive tasks. Together, these results highlight the central role of simulation in enabling AI agents to reason, plan, and act more effectively in the ever more challenging environments.
BibTeX
@inproceedings{yu2026dyna,
  author = {Xiao Yu and Baolin Peng and Michel Galley and Hao Cheng and Qianhui Wu and Janardhan Kulkarni and Suman Nath and Zhou Yu and Jianfeng Gao},
  title = {{Dyna-Mind: Learning to Simulate from Experience for Better AI Agents}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=F848aPzCJy},
}
Figure