When a human enters a new house looking for a kitchen, they don't wander randomly. They use a "mental model" of architecture—knowing that kitchens are often adjacent to dining rooms or located on the ground floor. Traditional robotic exploration, however, is often "blind" or "optimistic," treating unobserved areas as either empty voids or uniform obstacles until a sensor physically scans them.
A growing body of research is moving robots toward human-like intuition by treating spatial mapping as a generative inpainting task. Instead of just recording what they see, these agents "hallucinate" the unobserved parts of their environment to plan more efficient paths.
The Shift to Amodal Perception
The core idea is to move from modal perception (what is currently visible) to amodal perception (the complete structure, including occluded parts). In Occupancy Anticipation for Efficient Exploration and Navigation, Santhosh K. Ramakrishnan and Kristen Grauman demonstrate that agents can "see around corners" by predicting the layout of occluded regions using visual context.
By fusing semantic features from RGB images with geometric depth data, a UNet-based decoder can extrapolate local occupancy grids. This allows the robot to anticipate a hallway's continuation or a room's boundary before it even reaches the doorway. As Erik Wijmans and Dhruv Batra show in Seeing the Un-Scene, learning these architectural regularities allows agents to build amodal semantic maps that predict the location and shape of rooms beyond the current field of view, significantly outperforming standard reinforcement learning baselines.
Maps as Images: The Inpainting Mechanism
To a generative model, a 2D top-down map is essentially an image where pixels represent semantic labels (e.g., "road," "wall," "sofa"). This framing allows researchers to apply powerful computer vision techniques like Generative Adversarial Networks (GANs) and Diffusion Models to "fill in" the blanks.
Mark Campbell and Yutao Han pioneered this in Planning Paths Through Unknown Space by Imagining What Lies Therein, using GANs to predict semantic labels in occluded areas. By imagining the likely geometry of hidden structures—like extending a road segment or completing a building footprint—robots can avoid dead ends and reduce the need for frequent replanning.
More recently, the field has shifted toward Diffusion Models, which are better at capturing the complex, non-deterministic distributions of indoor spaces. In SePaint, Deepak Duggirala and Lantao Liu use multinomial diffusion to recover dense semantic understanding from sparse Bird’s-Eye-View (BEV) maps. Similarly, Kaijie Yun and colleagues introduced Diffusion as Reasoning (DAR), which conditions these hallucinations on common-sense knowledge from Large Language Models (LLMs) to predict where specific objects, like a "toilet" or "sofa," are likely to be located.
Why Hallucination Improves Planning
Why does "making things up" help a robot navigate? The answer lies in the cost function of path planning.
Traditional planners are reactive; they only see a wall when they are right in front of it. By contrast, a predictive system like ForesightNav, developed by Hardik Shah and Davide Scaramuzza, uses "scene imagination" to select smarter long-term goals. If the model predicts a high probability of a target object (like a bed) being behind a specific wall, the robot can plan a direct path toward that "hallucinated" goal rather than wandering aimlessly.
This approach is particularly effective in urban environments. Yutao Han and Youya Xia demonstrated in Planning Paths through Occlusions in Urban Environments that by imagining hidden parts of intersections, autonomous vehicles can plan smoother, more human-like paths much earlier than systems relying solely on visible LiDAR data.
Balancing Imagination with Uncertainty
The danger of a generative approach is "over-confidence"—a robot might plan a path through a hallucinated hallway that doesn't actually exist. To solve this, researchers use uncertainty-guided exploration.
In MapEx, Cherie Ho and Chen Wang use an ensemble of global map predictions to quantify uncertainty. The robot prioritizes "frontiers" where the internal model is least confident, effectively using its "imagination" to decide what it needs to verify next. Yizhuo Wang 0004 takes this further with CogniPlan, which generates multiple plausible structural hypotheses (e.g., "the corridor might turn left OR right") and uses graph-attention networks to reason about which path is most efficient across these different "dreams."
By treating the unknown not as a blank space, but as a puzzle to be solved using learned priors, these systems allow robots to navigate unfamiliar worlds with a level of foresight previously reserved for biological agents.