Imagine an agent navigating a simple 2D maze. Two positions, $A$ and $B$, sit on opposite sides of a thin, solid wall. In terms of raw spatial coordinates, $A$ and $B$ are nearly identical; their straight-line Euclidean distance is close to zero. However, to actually move from $A$ to $B$, the agent must traverse a long, circuitous corridor.
This discrepancy highlights the metric-topological gap: the fundamental mismatch between geometric distance (how close two states look in coordinate or pixel space) and topological distance (how difficult it actually is to transition between them given environmental constraints). When autonomous agents rely on standard geometric distance metrics, they suffer from Euclidean distance bias, leading to severe planning and learning failures.
The Danger of Obstacle-Blind Heuristics
Why does this geometric bias cause such widespread issues? Many reinforcement learning (RL) and motion planning algorithms rely on geometric distance as a surrogate for progress. When designing reward functions or generating subgoals, it is highly intuitive to reward an agent for minimizing its Euclidean distance to a target.
However, as Victor Campos 0001 et al. demonstrate in Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills, coordinate-based proximity metrics create deceptive gradients. An agent trying to reach a goal behind a wall receives high rewards for smashing directly into the barrier, trapping it in a local optimum. Similarly, Hyunki Seong et al. note in Skill Q-Network: Learning Adaptive Skill Ensemble for Mapless Navigation in Unknown Environments that over-optimistic Euclidean rewards mislead agents into getting stuck against obstacles rather than seeking alternative routes.
This metric bias breaks down across several core paradigms:
- Hindsight Goal Generation: In goal-conditioned RL, algorithms often pick intermediate subgoals from past trajectories to guide exploration. Matthias Brucker et al. show in Complex Robotic Manipulation via Graph-Based Hindsight Goal Generation (IEEE Trans. Neural Networks Learn. Syst. 2022) that Euclidean-based goal selection frequently proposes physically unreachable subgoals inside or behind obstacles, stalling the learning process.
- Data Augmentation and Offline RL: When stitching trajectories together in offline RL, identifying "similar" states using geometric metrics fails. As discussed by the authors of Offline Reinforcement Learning with Pseudometric Learning, states that are physically close may lead to entirely different outcomes due to obstacles, causing offline agents to hallucinate unsafe or impossible transitions.
- Kinodynamic Constraints: Even in obstacle-free spaces, geometric distance fails for non-holonomic systems (like a car with a minimum turning radius). As Hao-Tien Lewis Chiang et al. observe in RL-RRT: Kinodynamic Motion Planning via Learning Reachability Estimators From RL Policies, a target directly behind a vehicle is geometrically close but temporally distant, requiring a complex maneuvering loop that Euclidean metrics completely ignore.
The Solution: Learning Reachability-Based Latent Spaces
To bridge this gap, researchers focus on learning topology-aware latent representations. The core intuition is to warp the state space into a latent embedding where Euclidean distance directly corresponds to reachability or action effort.
If two states are separated by a wall, the embedding function should push them far apart in the latent space, effectively "stretching" the representation along the true path of traversability.
Physical Space (with obstacle) Warped Latent Space
[ State A ] [ State A ]
============== (Wall) |
[ State B ] | (True Path Distance)
|
[ State B ]
Mechanisms for Topological Mapping
How do we train an embedding function to respect topology without knowing the environment map beforehand? Several elegant mechanisms have emerged:
1. Approximating the Graph Laplacian
One foundational approach is to compute the eigenvectors of the graph Laplacian of the environment's transition dynamics. In The Laplacian in RL: Learning Representations with Efficient Approximations, Yifan Wu et al. show that these eigenvectors naturally preserve global geometry and connectivity. Mapping states into this Laplacian space provides a geometry-aware map where physical barriers are represented as significant latent distances. To scale this without global environment resets, Akram Erraqabi et al. introduce Temporal abstractions-augmented temporally contrastive learning: an alternative to the Laplacian in RL, which uses temporal contrastive learning to recover these topological properties from single starting points.
2. Actionable Representations (ARC)
Another intuition is to define distance by the effort required by the agent's policy. In Learning Actionable Representations with Goal-Conditioned Policies (and its subsequent work Learning Actionable Representations with Goal Conditioned Policies), Dibya Ghosh et al. introduce Actionable Representations for Control (ARC). ARC learns a latent space where the distance between two states matches the difference in the action sequences required to transition between them. If a wall requires a 50-step detour, the latent distance reflects those 50 steps, completely ignoring the 1-inch physical thickness of the wall.
3. Multi-Step Inverse Dynamics
Building on action-based distance, Anurag Koul et al. propose PCLAST in PcLast: Discovering Plannable Continuous Latent States. By pairing multi-step inverse dynamics with temporal contrastive objectives, they explicitly optimize for reachability, ensuring that states close in the latent space are guaranteed to be reachable from one another in the physical world.
4. Optimal Transport and Occupancy Matching
Rather than point-to-point metrics, some frameworks look at the geometry of entire trajectories. Thomas Rupf et al. introduce ZILOT in Zero-Shot Offline Imitation Learning via Optimal Transport, using Optimal Transport to match predicted state occupancies to expert paths, thereby respecting underlying physics and constraints. Similarly, Kai Yan utilizes the primal form of the Wasserstein distance in Offline Imitation from Observation via Primal Wasserstein State Occupancy Matching to capture environmental topology and prevent imitation learners from making short-sighted geometric errors.
Why It Works: The Intuition of Latent Warping
By shifting from coordinate-based metrics to reachability-based metrics, the agent's optimization landscape transforms. Deceptive gradients disappear because the "thin wall" is no longer thin in the latent space; it is a massive chasm.
When a hierarchical manager sets a subgoal, or a curriculum generator selects the next task (as in the diffusion-based curriculum proposed by Erdi Sayar et al. in Diffusion-based Curriculum Reinforcement Learning), the selection is naturally constrained to the manifold of traversable space. The agent plans efficiently because its internal geometry finally mirrors the physical reality of its environment.