Abstract
Goal-conditioned reinforcement learning (GCRL) requires agents to navigate to arbitrary goal states, where the optimal value function effectively represents the shortest path distance. Traditional GCRL methods often rely on Temporal Difference (TD) learning, which treats value functions as unconstrained black boxes and frequently suffers from instability and sample inefficiency. This paper introduces Quasimetric Reinforcement Learning (QRL), a framework that models value functions as quasimetrics—asymmetric distance functions that inherently satisfy the triangle inequality. By replacing standard Bellman updates with a geometric optimization objective that enforces local consistency while maximizing global state separation, QRL achieves state-of-the-art performance. Our evaluation across navigation and robotic manipulation tasks demonstrates that QRL improves sample efficiency by up to 4.9× and significantly outperforms existing baselines in offline settings.
Introduction
In goal-conditioned reinforcement learning (GCRL), an agent's objective is to learn a policy that can reach any arbitrary goal state $g$ from a starting state $s$. The "cost-to-go," or the negative value function, represents the distance between these states. Despite the success of deep RL, current approaches face two primary hurdles. First, they often ignore the underlying geometry of the task; optimal value functions in GCRL are not arbitrary but must satisfy the triangle inequality ($dist(a, c) \le dist(a, b) + dist(b, c)$). Second, the reliance on Temporal Difference (TD) learning—originally designed for unconstrained function spaces—can lead to slow convergence and instability when applied to the restricted geometric spaces of GCRL.
This paper proposes Quasimetric Reinforcement Learning (QRL) to address these inefficiencies. By treating the value function as a quasimetric, QRL leverages the inherent structure of the problem to learn more robust and accurate representations of the environment's dynamics.
Methodology
Quasimetrics as Value Functions
A quasimetric is a distance function that satisfies the standard properties of a metric (identity of indiscernibles and the triangle inequality) but allows for asymmetry. In many real-world environments, the cost to move from state $A$ to $B$ is not equal to the cost from $B$ to $A$—consider walking up a steep hill versus walking down, or navigating a one-way street. The authors establish a "Value-Quasimetric Equivalence," proving that the set of all possible optimal value functions in GCRL corresponds exactly to the set of all quasimetrics.
The QRL Objective
Rather than relying on iterative bootstrapping via TD learning, QRL employs a direct geometric optimization. The authors use a physical analogy: imagine states as points connected by chains representing local transitions. If one pulls the start and goal states as far apart as possible, the chains eventually tighten. At this point, the distance between the two states equals the shortest path.
QRL implements this via two primary components:
- Local Constraint: It ensures the learned distance $d(s, s')$ does not exceed the actual cost of a single transition observed in the data.
- Global Objective: It maximizes the distance $d(s, g)$ between all pairs of states in the dataset, effectively "stretching" the learned geometry until it hits the limits imposed by the local constraints.
Interval Quasimetric Embeddings (IQE)
To ensure the learned function remains a valid quasimetric throughout training, the authors introduce Interval Quasimetric Embeddings (IQE). This neural network architecture is designed to satisfy the triangle inequality by construction. By enforcing this geometric constraint within the architecture itself, QRL avoids the need for complex penalty terms and maintains a consistent search space.
Experimental Evaluation
The authors evaluated QRL across a variety of challenging environments, comparing it against standard TD-based and contrastive RL baselines.
Online Robotic Manipulation
In the Fetch Robot benchmarks (FetchReach, FetchPush, and FetchSlide), QRL demonstrated remarkable sample efficiency. In the FetchPush environment, QRL achieved a 4.9× improvement in sample efficiency over Contrastive RL. This suggests that the geometric inductive bias allows the agent to understand environment dynamics much faster than black-box models.
Offline Navigation
Using the D4RL maze2d datasets, the authors tested QRL's ability to learn from static, sub-optimal data. QRL outperformed the best-performing baselines by over 37% and even surpassed hand-coded reference controllers by 46%. These results highlight QRL's ability to extract optimal paths even when the training data is noisy or limited.
Planning and Search
Because the learned quasimetric is "admissible" (it never overestimates the true cost), it can be used as a heuristic for classical search algorithms. When integrated with trajectory samplers like the Diffuser model, QRL's value function significantly improved success rates in multi-goal settings, providing better guidance than standard value functions.
Discussion and Future Work
QRL represents a significant shift in how value functions are conceptualized and trained. By acknowledging the asymmetric nature of directed dynamics, it captures environmental constraints that symmetric metrics miss. However, the current framework primarily assumes deterministic dynamics. While it has shown success in image-based tasks with implicit stochasticity, extending the theoretical guarantees to fully stochastic Markov Decision Processes (MDPs) is a key area for future research.
Additionally, the learned geometric structure offers new avenues for exploration. Instead of standard epsilon-greedy strategies, future iterations of QRL could utilize "geometry-aware" exploration, where the agent actively seeks out transitions in areas where the quasimetric structure is most uncertain.
Conclusion
Quasimetric Reinforcement Learning provides a principled geometric approach to goal-conditioned RL. By constraining value functions to the space of quasimetrics and utilizing a direct optimization objective, QRL overcomes the traditional bottlenecks of TD learning. The result is a more stable, sample-efficient, and accurate method for learning to reach goals in complex, asymmetric environments.
Summary updated