This paper presents RIWANav, a framework for urban navigation that uses a "recursive self-improvement" loop to co-evolve a navigation policy and a visual world model. By allowing the agent to learn from "imagined" failures and then refining the world model to match the agent's evolving behavior, RIWANav achieves state-of-the-art performance in complex city environments without the need for costly or dangerous real-world trial-and-error.
TL;DR
RIWANav introduces a recursive training loop that co-evolves a navigation policy and a visual world model. By using the world model to "imagine" consequences for policy training and then using the improved policy to refine the world model's predictions, it achieves significant performance gains in urban navigation without requiring physical trial-and-error.
What Problem Does This Paper Address?
Long-horizon urban navigation—getting from point A to point B in a city based on visual cues and language instructions—is notoriously difficult because small local errors compound over time.
Standard Imitation Learning (IL) is the go-to starting point, but it has a major flaw: it only learns from successful expert trajectories. It never sees "failures," so it doesn't know how to recover when it inevitably drifts off-course. Reinforcement Learning (RL) can learn from failures, but running physical robots until they crash in a busy city is expensive, slow, and dangerous.
Action-conditioned World Models (WM) offer a solution by acting as a "mental simulator" where an agent can imagine the results of its actions. However, if the world model stays "frozen" while the policy improves, it eventually becomes a bottleneck. As the policy explores new behaviors, the frozen world model's predictions become inaccurate, providing poor feedback for further learning.
Methods
The core of RIWANav is a Recursive Self-Improvement (RSI) loop that alternates between two coupled phases:
- Policy Improvement: The agent uses the current world model to evaluate multiple candidate actions from the same state. By comparing the "imagined" visual outcomes (e.g., "does this action lead me closer to the landmark?"), the policy is optimized using Group Relative Policy Optimization (GRPO). This allows the agent to learn from imagined failures in a safe, simulated sandbox.
- World Model Improvement: The improved policy then generates new action proposals. To ensure the world model learns from valid data, these are filtered for "expert consistency" (ensuring they stay relatively close to real recorded motion). A grounded self-curriculum then prioritizes these samples based on behavioral novelty (how much they differ from the starting policy) and prediction error. This forces the world model to adapt specifically to the new ways the policy is choosing to move.
Datasets
The authors evaluate RIWANav on the URBANNAV dataset, a large-scale collection of city-scale human trajectories aligned with landmark-grounded language instructions. They test on "seen" environments (used during training) and "unseen" environments to measure generalization.
Key Insights and Ideas
- Co-Evolution is Essential: A world model is only useful if it is accurate for the current policy. The "recursive" nature of RIWANav ensures that as the policy gets smarter and explores new state-action spaces, the simulator follows, providing high-fidelity feedback.
- Imagination as a Safety Buffer: By using a world model, the agent can "fail" thousands of times in its head to learn what not to do. This bridges the gap between passive imitation and active, risky exploration.
- Grounded Curriculum: Instead of training on all available data, the system focuses on "hard" and "novel" samples. This targeted refinement prevents the world model from drifting and ensures it remains a reliable teacher for the next policy update.
Key Results
- Performance Boost: On unseen urban environments, RIWANav achieved an 88.25% Success Rate (SR), a massive jump from the 67.88% of standard imitation learning and significantly better than the 80.31% achieved with a frozen world model.
-
Fidelity Gains: The refined world model showed a marked improvement in visual prediction quality. As shown in Figure 3, both pixel-level (L1) and perceptual (LPIPS) errors decreased steadily across recursive cycles. World model prediction quality improves over recursive rounds for both expert and policy-induced actions.
-
Visual Coherence: Qualitatively, the updated world model preserves scene structure—like sidewalk boundaries and vehicle arrangements—much better during long rollouts compared to the initial model, which often "lost" the scene structure over time. Qualitative comparison showing RIWANav's world model produces more coherent and traversable "imagined" futures.
-
Real-World Transfer: The authors successfully deployed the trained policy on a physical mobile robot, proving that the "imagined" training translates effectively to real-world city navigation.
Limitations and Future Work
- Data Dependency: The world model refinement still relies on a fixed offline dataset for ground-truth future frames. This limits how far the policy can "explore" away from expert-like behavior while still receiving valid supervision.
- Computational Cost: Training and recursively updating large video-based world models is computationally intensive, requiring significant GPU resources (the authors used four NVIDIA H200s).
- Generalization: While successful in the tested urban environments, broader evaluations across more diverse cities and different robot embodiments are needed to confirm the framework's universal applicability.
Summary updated