Imagine a robot navigating a complex maze. Two positions separated by a thin wall are physically inches apart, and their camera views might look nearly identical. Yet, to travel from one side of the wall to the other, the robot must navigate the entire length of the maze.
Standard spatial metrics—like Euclidean distance in pixel space or coordinate space—fail catastrophically in these scenarios because they ignore environmental constraints. As Bobrin et al. note, physical or visual adjacency does not imply ease of navigation.
To solve this, a prominent direction in reinforcement learning and robotics focuses on learning temporal distance embeddings. The core idea is to map high-dimensional observations into a latent space where the geometric distance between two points directly reflects their temporal reachability—the minimum number of environment steps or actions required to transition from one state to another. By embedding time into geometry, complex planning and decision-making problems collapse into straightforward vector operations.
Mapping Time to Geometry via Contrastive Learning
How do we force a neural network to understand the topology of an environment using only raw data? The most common mechanism leverages temporal contrastive learning over observed trajectories. By treating time as a supervisor, the model minimizes the latent distance between states that occur close together in time while pushing temporally distant or randomly sampled states apart.
[State s_t] --(Close in time)--> [State s_{t+k}] ==> Minimize Latent Distance
[State s_t] --(Far in time)----> [State s_random] ==> Maximize Latent Distance
Evans demonstrates this by utilizing a temporal contrastive objective to transform continuous latent states into a metric space where Euclidean distance directly approximates reachability. This approach effectively embeds the environment's global topology into local latent geometry without requiring a hand-crafted map.
Similarly, Campbell et al. show that organizing state spaces by temporal reachability rather than visual similarity allows agents to group states that are functionally close. This self-supervised organization can also be enforced using triplet losses. For example, Wu et al. apply this to multimodal robotic observations (pixels and tactile feedback) to map task progress, while Zanjani et al. use it to enforce temporal consistency for cross-modal indoor localization.
Overcoming the Asymmetry of Time
A pure Euclidean metric space assumes symmetry: the distance from point $A$ to point $B$ is identical to the distance from $B$ to $A$. However, physical environments are inherently directed. Walking down a steep hill is trivial; climbing back up takes significant effort. Dropping a glass breaks it instantly, but reversing that state is impossible.
To capture these non-symmetric dynamics, researchers have moved beyond standard symmetric distance metrics:
- Asymmetric Norms: The authors of Asymmetric Norms to Approximate the Minimum Action Distance introduce asymmetric "Wide Norms" to train embeddings where the distance metric serves as a direct, directional heuristic for reachability.
- Dual/Bilinear Representations: Another approach is to decouple the representations of current states and target goals. In Dual Goal Representations, the complex, non-symmetric temporal distance function is approximated using a learned inner product between separate state and goal embedding vectors. This bilinear form provides the expressivity needed to capture complex reachability relationships without forcing a symmetric geometric constraint.
Why It Works: Downstream Applications
Structuring a latent space around temporal distance provides several fundamental advantages for control, planning, and exploration.
1. Trajectory Stitching and Hierarchical Planning
When the latent space reflects reachability, long-horizon tasks can be broken down into a sequence of reachable waypoints. Lee et al. construct navigation graphs where edges are defined by learned temporal distances rather than physical proximity, allowing agents to bypass obstacles seamlessly. For offline reinforcement learning, Baek et al. use this geometric consistency to filter noisy data and "stitch" fragmented, sub-optimal trajectories into optimal global paths.
2. Zero-Shot Generalization and Foundation Models
By treating reachability as a universal coordinate system, agents can generalize to entirely new tasks without retraining. Park et al. map states into a latent space where navigation is treated as simple vector-based movement, enabling zero-shot capabilities.
On a massive scale, the framework presented in RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance demonstrates that the literal time remaining to reach a goal serves as a physically grounded, universal reward metric. Because time maintains a consistent physical meaning across different robot embodiments and task durations, it eliminates the need for complex cross-task calibration.
3. Robust Exploration
Standard exploration algorithms often rely on visual novelty, which fails when superficial changes (like a flickering screen or shifting shadows) trick the agent into thinking it has found a new state.
In contrast, the authors of Time-Myopic Go-Explore: Learning A State Representation for the Go-Explore Paradigm show that using learned temporal distance as a novelty metric prevents agents from conflating visually similar but functionally distinct areas. The agent builds a stable archive of milestones based on how long it takes to transition between states, leading to highly robust exploration in sparse-reward environments.
Key Foundational Work
Self-supervised Learning of Image Embedding for Continuous Control (2019) — contributed 2 concept elements to this direction.