Overcoming the Metric Topological Gap in Reinforcement Learning

238 concepts174 papers5 proposals

Imagine an agent navigating a simple 2D maze. Two positions, $A$ and $B$, sit on opposite sides of a thin, solid wall. In terms of raw spatial coordinates, $A$ and $B$ are nearly identical; their straight-line Euclidean distance is close to zero. However, to actually move from $A$ to $B$, the agent must traverse a long, circuitous corridor.

This discrepancy highlights the metric-topological gap: the fundamental mismatch between geometric distance (how close two states look in coordinate or pixel space) and topological distance (how difficult it actually is to transition between them given environmental constraints). When autonomous agents rely on standard geometric distance metrics, they suffer from Euclidean distance bias, leading to severe planning and learning failures.

The Danger of Obstacle-Blind Heuristics

Why does this geometric bias cause such widespread issues? Many reinforcement learning (RL) and motion planning algorithms rely on geometric distance as a surrogate for progress. When designing reward functions or generating subgoals, it is highly intuitive to reward an agent for minimizing its Euclidean distance to a target.

However, as Victor Campos 0001 et al. demonstrate in Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills, coordinate-based proximity metrics create deceptive gradients. An agent trying to reach a goal behind a wall receives high rewards for smashing directly into the barrier, trapping it in a local optimum. Similarly, Hyunki Seong et al. note in Skill Q-Network: Learning Adaptive Skill Ensemble for Mapless Navigation in Unknown Environments that over-optimistic Euclidean rewards mislead agents into getting stuck against obstacles rather than seeking alternative routes.

This metric bias breaks down across several core paradigms:

The Solution: Learning Reachability-Based Latent Spaces

To bridge this gap, researchers focus on learning topology-aware latent representations. The core intuition is to warp the state space into a latent embedding where Euclidean distance directly corresponds to reachability or action effort.

If two states are separated by a wall, the embedding function should push them far apart in the latent space, effectively "stretching" the representation along the true path of traversability.

Physical Space (with obstacle)           Warped Latent Space
        [ State A ]                              [ State A ]
      ============== (Wall)                           |
        [ State B ]                                   | (True Path Distance)
                                                      |
                                                 [ State B ]

Mechanisms for Topological Mapping

How do we train an embedding function to respect topology without knowing the environment map beforehand? Several elegant mechanisms have emerged:

1. Approximating the Graph Laplacian

One foundational approach is to compute the eigenvectors of the graph Laplacian of the environment's transition dynamics. In The Laplacian in RL: Learning Representations with Efficient Approximations, Yifan Wu et al. show that these eigenvectors naturally preserve global geometry and connectivity. Mapping states into this Laplacian space provides a geometry-aware map where physical barriers are represented as significant latent distances. To scale this without global environment resets, Akram Erraqabi et al. introduce Temporal abstractions-augmented temporally contrastive learning: an alternative to the Laplacian in RL, which uses temporal contrastive learning to recover these topological properties from single starting points.

2. Actionable Representations (ARC)

Another intuition is to define distance by the effort required by the agent's policy. In Learning Actionable Representations with Goal-Conditioned Policies (and its subsequent work Learning Actionable Representations with Goal Conditioned Policies), Dibya Ghosh et al. introduce Actionable Representations for Control (ARC). ARC learns a latent space where the distance between two states matches the difference in the action sequences required to transition between them. If a wall requires a 50-step detour, the latent distance reflects those 50 steps, completely ignoring the 1-inch physical thickness of the wall.

3. Multi-Step Inverse Dynamics

Building on action-based distance, Anurag Koul et al. propose PCLAST in PcLast: Discovering Plannable Continuous Latent States. By pairing multi-step inverse dynamics with temporal contrastive objectives, they explicitly optimize for reachability, ensuring that states close in the latent space are guaranteed to be reachable from one another in the physical world.

4. Optimal Transport and Occupancy Matching

Rather than point-to-point metrics, some frameworks look at the geometry of entire trajectories. Thomas Rupf et al. introduce ZILOT in Zero-Shot Offline Imitation Learning via Optimal Transport, using Optimal Transport to match predicted state occupancies to expert paths, thereby respecting underlying physics and constraints. Similarly, Kai Yan utilizes the primal form of the Wasserstein distance in Offline Imitation from Observation via Primal Wasserstein State Occupancy Matching to capture environmental topology and prevent imitation learners from making short-sighted geometric errors.

Why It Works: The Intuition of Latent Warping

By shifting from coordinate-based metrics to reachability-based metrics, the agent's optimization landscape transforms. Deceptive gradients disappear because the "thin wall" is no longer thin in the latent space; it is a massive chasm.

When a hierarchical manager sets a subgoal, or a curriculum generator selects the next task (as in the diffusion-based curriculum proposed by Erdi Sayar et al. in Diffusion-based Curriculum Reinforcement Learning), the selection is naturally constrained to the manifold of traversable space. The agent plans efficiently because its internal geometry finally mirrors the physical reality of its environment.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_euclidean_distance_bias_2bc05494,
  title = {Euclidean distance bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/euclidean-distance-bias/rdr_2bc05494890b4025aa50f0ff1a15e848},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_euclidean_distance_bias_2bc05494_v1,
  title = {Euclidean distance bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/euclidean-distance-bias/rdr_2bc05494890b4025aa50f0ff1a15e848/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake