Abstract
People navigating in unfamiliar buildings take advantage of myriad visual, spatial and semantic cues to efficiently achieve their navigation goals. Towards equipping computational agents with similar capabilities, we introduce Pathdreamer, a visual world model for agents navigating in novel indoor environments. Given one or more previous visual observations, Pathdreamer generates plausible high-resolution 360° visual observations (RGB, semantic segmentation and depth) for viewpoints that have not been visited, in buildings not seen during training. In regions of high uncertainty (e.g. predicting around corners, imagining the contents of an unseen room), Pathdreamer can predict diverse scenes, allowing an agent to sample multiple realistic outcomes for a given trajectory. We demonstrate that Pathdreamer encodes useful and accessible visual, spatial and semantic knowledge about human environments by using it in the downstream task of Vision-and-Language Navigation (VLN). Specifically, we show that planning ahead with Pathdreamer brings about half the benefit of looking ahead at actual observations from unobserved parts of the environment.
Introduction
When humans enter a novel environment, they rely on spatial intuition to predict what lies beyond their immediate field of view. For instance, observing a tiled floor and a refrigerator leads one to expect a kitchen rather than a bedroom. Computational agents often lack this common-sense spatial reasoning, which is critical for efficient navigation. Pathdreamer addresses this by tackling the "room reveal" problem: generating photorealistic and semantically consistent 360° observations for unvisited locations in complex indoor environments.
Methodology
Pathdreamer utilizes a hierarchical, two-stage architecture to synthesize visual sequences along a proposed navigation path. The model maintains a persistent 3D point cloud as a memory of the environment's geometry and appearance to ensure long-term consistency.
Stage 1: Structure Generator
The first stage predicts the underlying structure of the scene—specifically semantic segmentation and depth maps. It re-projects previous observations from the 3D point cloud into the agent's new viewpoint. To handle the inherent uncertainty of unobserved areas, this stage is stochastic; it uses a latent noise tensor to sample multiple plausible layouts, allowing the agent to "dream" different potential futures for the same path.
Stage 2: Image Generator
The second stage translates the predicted structure into high-resolution RGB images. It employs Spatially-Adaptive Normalization (SPADE) blocks to paint realistic textures onto the semantic scaffold. This hierarchical approach—predicting semantics before pixels—is crucial, as it allows the model to leverage high-level environmental regularities (e.g., cabinets appearing near counters).
Experimental Setup and Datasets
The model was trained using the Matterport3D dataset, which contains over 10,000 high-resolution panoramas across 90 building-scale environments. Training data for the Structure Generator was rendered using the Habitat Simulator. Evaluation was conducted on the Room-to-Room (R2R) dataset, focusing on the model's ability to generalize to buildings not seen during training.
Results and Discussion
Pathdreamer demonstrates significant improvements in visual quality over baseline models, maintaining reasonable accuracy for trajectories up to 13 meters. In the downstream task of Vision-and-Language Navigation (VLN), an agent using a 3-step "look-ahead" with Pathdreamer achieved a 50.4% success rate, a substantial increase from the 40.6% baseline. This indicates that Pathdreamer captures approximately half the total benefit of having access to actual ground-truth observations of unvisited areas.
Despite these successes, limitations remain. The model's predictions tend to drift or blur over very long trajectories (beyond 10 meters), and it struggles with intricate geometries like stair railings. Furthermore, the current implementation assumes a static environment and operates on discrete navigation graphs.
Conclusion
Pathdreamer represents a significant step toward equipping robotic agents with human-like spatial imagination. By enabling mental simulations of unobserved spaces, it provides a powerful tool for model-based approaches to challenging embodied navigation tasks.
Summary updated