Abstract
We introduce a new reinforcement learning approach combining a planning quasi-metric (PQM) that estimates the number of steps required to go from any state to another, with task-specific "aimers" that compute a target state to reach a given goal. This decomposition allows the sharing across tasks of a task-agnostic model of the quasi-metric that captures the environment's dynamics and can be learned in a dense and unsupervised manner. We achieve multiple-fold training speed-up compared to recently published methods on the standard bit-flip problem and in the MuJoCo robotic arm simulator.
Introduction
Traditional Reinforcement Learning (RL) agents often struggle with generalization because their learned policies entangle environmental dynamics with specific task objectives. When a task changes, the agent typically restarts learning from scratch, failing to leverage its existing "understanding" of the world's physics. Furthermore, sparse reward signals in many environments lead to inefficient exploration and slow convergence.
This research addresses these challenges by decoupling the learning of environment dynamics from task-specific goals. By separating "how the world works" from "what needs to be achieved," the authors enable efficient transfer learning and utilize dense, unsupervised feedback to accelerate training.
Methods: PQM and Aimers
The proposed architecture consists of two primary components:
1. Planning Quasi-Metric (PQM)
The PQM is a learned model $f(s, s', a)$ that estimates the minimum number of steps required to transition from state $s$ to state $s'$, starting with action $a$. Unlike standard metrics, a quasi-metric accounts for the inherent asymmetry in many environments (e.g., moving uphill vs. downhill). Crucially, the PQM is task-agnostic; it focuses solely on the transition dynamics of the environment. It is trained using a Bellman-like consistency loss, providing a dense training signal from every transition experienced by the agent, regardless of the current task.
2. Aimers
Aimers are lightweight, task-specific models $h(s, g)$ that bridge high-level goals and low-level planning. Given a current state $s$ and a goal $g$ (which may represent a set of states), the aimer identifies the optimal "target state" $s'$ within that goal set that is easiest to reach according to the PQM.
Workflow: To execute an action, the Aimer selects a Target State based on the goal. The agent then utilizes the PQM (acting as a critic) and an associated Actor to select the action that most effectively reduces the distance to that target state.
Experimental Setup
The framework was evaluated on two distinct benchmarks:
- Bit-Flip Problem: A mathematical environment requiring the agent to match a target binary configuration. It features an exponential state space and sparse rewards.
- MuJoCo (Fetch Environments): A high-fidelity physics simulator used for robotic tasks, specifically "Push" (moving a block) and "Pick and Place" (lifting and moving a block).
Results
The PQM approach demonstrated significant performance gains across both benchmarks.
Training Speedup and Transfer
In the 30-bit flip environment, the PQM approach combined with transfer learning achieved a 100% success rate nearly an order of magnitude faster than standard Deep Q-Learning (DQN).
In MuJoCo, the benefits of transfer learning were even more pronounced. When transferring knowledge from a "Pick and Place" task to a "Push" task, the PQM-based agent exhibited a massive "jump-start," reaching high success rates almost immediately. In contrast, baseline models like DDPG+HER required significantly more training epochs to achieve comparable performance.
Model Accuracy
Analysis of the learned PQM confirmed that it accurately reflects true distances within the environment—such as Hamming distance in the bit-flip task or physical steps in MuJoCo. This indicates that the model successfully captures the underlying structural dynamics of the world in a task-agnostic manner.
Conclusion
By isolating world dynamics into a Planning Quasi-Metric, this framework allows the "heavy lifting" of environmental understanding to be performed once and reused across multiple tasks. While the PQM is more complex to train initially than a standard policy, its ability to provide dense unsupervised feedback and facilitate rapid transfer learning offers a powerful path forward for complex RL problems. Future work may explore hierarchical planning using aimers to set intermediate waypoints and leveraging the PQM for curiosity-driven exploration.
Summary updated