Abstract
Goal-Conditioned Reinforcement Learning (GCRL) often struggles with the dual challenges of high-dimensional state-goal spaces and sparse reward signals. This paper introduces a self-supervised method to learn state representations where the distance between embedded states in a latent space directly approximates the Minimum Action Distance (MAD)—the fewest number of actions required to transition between them. By leveraging offline, unlabeled trajectory data, this approach enables efficient latent-space planning and effective reward shaping, significantly improving performance in environments where rewards are infrequent or exploration is difficult.
Introduction
In GCRL, an agent is tasked with reaching various goal states within an environment. Traditional approaches typically utilize Universal Value Functions, which take both the current state and the goal state as inputs. However, this formulation leads to a quadratic increase in input complexity, making the learning task significantly more difficult.
Furthermore, many RL environments are characterized by sparse rewards, where the agent receives feedback only upon reaching the goal. Without a meaningful metric to gauge proximity to the goal, exploration becomes a "needle in a haystack" problem. While manual reward engineering can mitigate this, it requires domain expertise and is often not scalable. This research proposes a self-supervised framework to learn a meaningful distance metric directly from raw experience, providing a "compass" for the agent without human intervention.
Methodology
The core contribution of this work is a novel training scheme for a state embedding function, $\phi_\theta$, which maps raw states into a 64-dimensional latent space.
Minimum Action Distance (MAD)
The authors define the MAD as the shortest path (in terms of discrete steps) between two states. Since the true MAD is generally unknown in complex environments, the framework utilizes Trajectory Distance (TD)—the observed number of steps between two states in a recorded sequence—as an empirical upper bound.
Constrained Optimization
The embedding is trained using a neural network that minimizes the difference between the $L_1$ distance of embedded states and their observed trajectory distance. Crucially, the loss function enforces a constraint: the embedded distance must not exceed the trajectory distance. By minimizing the distance while respecting this upper bound, the network is "pushed" to find the shortest possible path, effectively approximating the MAD. The authors specifically utilize the $L_1$ norm (Manhattan distance) over the $L_2$ norm, as it is more robust in high-dimensional spaces and less susceptible to the "curse of dimensionality."
Transition and Planning
Once the embedding space is established, a transition model $\rho_\zeta$ is trained to predict the next latent state given a current embedding and an action. This enables two primary applications:
- Plan-Dist Algorithm: A Model Predictive Control (MPC) approach where the agent simulates action sequences in the latent space and selects the path that minimizes the distance to the goal embedding.
- Potential-Based Reward Shaping (PBRS): The learned distance is used as a heuristic to provide dense, intermediate rewards, signaling to the agent whether it is moving closer to or further from its objective.
Experimental Evaluation
The authors evaluated the MAD framework across several benchmarks, including classic control tasks (MountainCar, CartPole, AcroBot, Pendulum) and multi-goal environments (GridWorld and the Sawyer robotic arm).
Key Findings
- Superior Planning: In multi-goal tasks like the Sawyer robotic arm, the Plan-Dist algorithm outperformed Goal-Conditioned Supervised Learning (GCSL), reaching goals with higher precision and following more efficient paths.
- Solving Sparse Rewards: In the MountainCar environment, which is notoriously difficult due to its sparse reward structure, the addition of the learned distance metric as a reward signal (DDQN-PR) allowed agents to learn significantly faster than standard DDQN baselines.
- Robustness to Data Quality: The method proved capable of learning useful representations even from suboptimal data, such as random-walk trajectories in a GridWorld. This indicates that the model does not require expert demonstrations to understand the underlying geometry of the environment.
Discussion and Future Work
While the MAD framework provides a powerful tool for GCRL, it currently operates under the assumption of symmetry—that the distance from state A to B is identical to the distance from B to A. This assumption does not hold in environments with one-way transitions or directed constraints. Future research may explore asymmetric distance functions to accommodate such Directed Acyclic Graph (DAG) structures.
Additionally, in highly stochastic environments, the "minimum" distance may be less informative than the "expected" distance. The authors suggest that incorporating a weighted objective function could better handle environmental randomness. Finally, integrating this distance metric into Hierarchical Reinforcement Learning (HRL) could allow high-level agents to set more effective intermediate subgoals for low-level controllers.
Summary updated