RIWANav: Recursive World-Action Models with Self-Improvement for Urban Navigation

Jing Xie, Shouwei Ruan, Yubin Wang, Yuxiang Zhang, Haitao Yang, Songchang Jin, Dianxi ShiarXiv 2026 · arXiv

This paper presents RIWANav, a framework for urban navigation that uses a "recursive self-improvement" loop to co-evolve a navigation policy and a visual world model. By allowing the agent to learn from "imagined" failures and then refining the world model to match the agent's evolving behavior, RIWANav achieves state-of-the-art performance in complex city environments without the need for costly or dangerous real-world trial-and-error.

TL;DR

RIWANav introduces a recursive training loop that co-evolves a navigation policy and a visual world model. By using the world model to "imagine" consequences for policy training and then using the improved policy to refine the world model's predictions, it achieves significant performance gains in urban navigation without requiring physical trial-and-error.

What Problem Does This Paper Address?

Long-horizon urban navigation—getting from point A to point B in a city based on visual cues and language instructions—is notoriously difficult because small local errors compound over time.

Standard Imitation Learning (IL) is the go-to starting point, but it has a major flaw: it only learns from successful expert trajectories. It never sees "failures," so it doesn't know how to recover when it inevitably drifts off-course. Reinforcement Learning (RL) can learn from failures, but running physical robots until they crash in a busy city is expensive, slow, and dangerous.

Action-conditioned World Models (WM) offer a solution by acting as a "mental simulator" where an agent can imagine the results of its actions. However, if the world model stays "frozen" while the policy improves, it eventually becomes a bottleneck. As the policy explores new behaviors, the frozen world model's predictions become inaccurate, providing poor feedback for further learning.

Methods

The core of RIWANav is a Recursive Self-Improvement (RSI) loop that alternates between two coupled phases:

  1. Policy Improvement: The agent uses the current world model to evaluate multiple candidate actions from the same state. By comparing the "imagined" visual outcomes (e.g., "does this action lead me closer to the landmark?"), the policy is optimized using Group Relative Policy Optimization (GRPO). This allows the agent to learn from imagined failures in a safe, simulated sandbox.
  2. World Model Improvement: The improved policy then generates new action proposals. To ensure the world model learns from valid data, these are filtered for "expert consistency" (ensuring they stay relatively close to real recorded motion). A grounded self-curriculum then prioritizes these samples based on behavioral novelty (how much they differ from the starting policy) and prediction error. This forces the world model to adapt specifically to the new ways the policy is choosing to move.
The RIWANav recursive loop alternates between policy optimization via imagined feedback and world model refinement via a policy-induced curriculum.

Datasets

The authors evaluate RIWANav on the URBANNAV dataset, a large-scale collection of city-scale human trajectories aligned with landmark-grounded language instructions. They test on "seen" environments (used during training) and "unseen" environments to measure generalization.

Key Insights and Ideas

  • Co-Evolution is Essential: A world model is only useful if it is accurate for the current policy. The "recursive" nature of RIWANav ensures that as the policy gets smarter and explores new state-action spaces, the simulator follows, providing high-fidelity feedback.
  • Imagination as a Safety Buffer: By using a world model, the agent can "fail" thousands of times in its head to learn what not to do. This bridges the gap between passive imitation and active, risky exploration.
  • Grounded Curriculum: Instead of training on all available data, the system focuses on "hard" and "novel" samples. This targeted refinement prevents the world model from drifting and ensures it remains a reliable teacher for the next policy update.

Key Results

  • Performance Boost: On unseen urban environments, RIWANav achieved an 88.25% Success Rate (SR), a massive jump from the 67.88% of standard imitation learning and significantly better than the 80.31% achieved with a frozen world model.
  • Fidelity Gains: The refined world model showed a marked improvement in visual prediction quality. As shown in Figure 3, both pixel-level (L1) and perceptual (LPIPS) errors decreased steadily across recursive cycles. World model prediction quality improves over recursive rounds for both expert and policy-induced actions.

  • Visual Coherence: Qualitatively, the updated world model preserves scene structure—like sidewalk boundaries and vehicle arrangements—much better during long rollouts compared to the initial model, which often "lost" the scene structure over time. Qualitative comparison showing RIWANav's world model produces more coherent and traversable "imagined" futures.

  • Real-World Transfer: The authors successfully deployed the trained policy on a physical mobile robot, proving that the "imagined" training translates effectively to real-world city navigation.

World model prediction quality improves over recursive rounds for both expert and policy-induced actions.
Qualitative comparison showing RIWANav's world model produces more coherent and traversable "imagined" futures.

Limitations and Future Work

  • Data Dependency: The world model refinement still relies on a fixed offline dataset for ground-truth future frames. This limits how far the policy can "explore" away from expert-like behavior while still receiving valid supervision.
  • Computational Cost: Training and recursively updating large video-based world models is computationally intensive, requiring significant GPU resources (the authors used four NVIDIA H200s).
  • Generalization: While successful in the tested urban environments, broader evaluations across more diverse cities and different robot embodiments are needed to confirm the framework's universal applicability.

Summary updated

Abstract

Long-horizon urban navigation requires sequential local decisions whose errors can compound over time. Imitation learning (IL) rarely learns from failures, while physical trial-and-error reinforcement learning (RL) is costly. Action-conditioned world models can provide imagined feedback by predicting visual consequences for candidate actions. However, a frozen world model may become less reliable as the policy evolves. In this paper, we introduce RIWANAV, a post-training framework that casts the coupled adaptation of a world model and an action model (policy) as task-specific recursive self-improvement (RSI). Each cycle alternates two updates. The world model evaluates policy actions through imagined outcomes, providing comparative feedback for group-relative policy optimization (GRPO). The improved policy then constructs a grounded self-curriculum, selecting expert-consistent action-video pairs by behavioral novelty and prediction error. The refined world model supplies feedback for the next policy update, closing the recursive self-improvement loop. Experiments show that RIWANAV outperforms training baselines and prior methods, validating the proposed recursive self-improvement loop between the policy and world model. Real-world trials further demonstrate its practical applicability.
BibTeX
@misc{xie2026riwanav,
  author = {Jing Xie and Shouwei Ruan and Yubin Wang and Yuxiang Zhang and Haitao Yang and Songchang Jin and Dianxi Shi},
  title = {{RIWANav: Recursive World-Action Models with Self-Improvement for Urban Navigation}},
  year = {2026},
  eprint = {2610.08640},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.08640},
}
Figure