Abstract
Offline reinforcement learning (RL) typically struggles with distribution shift and sub-optimal data. Traditional methods often rely on behavior cloning to stay within the data distribution, which inadvertently forces the agent to mimic poor demonstrations. This paper introduces Partial Policy Learning (PPL), a framework that reformulates offline RL as a Partial Optimal Transport (POT) problem. By treating the Q-function as a transport cost, PPL enables selective "stitching" of high-quality behaviors from diverse datasets while ignoring sub-optimal noise. Experimental results across D4RL benchmarks demonstrate that PPL achieves state-of-the-art performance, particularly in complex navigation and manipulation tasks.
Introduction
Offline reinforcement learning is the challenge of deriving an optimal policy from a fixed, pre-collected dataset without further environment interaction. The primary obstacle in this setting is distribution shift: when an agent selects an action not well-represented in the training data, the value function (critic) often produces overoptimistic estimates, leading to policy failure.
To mitigate this, many existing algorithms employ Behavior Cloning (BC) or distribution constraints to keep the learned policy close to the dataset. However, real-world datasets are rarely perfect; they often contain a mix of expert demonstrations, mediocre attempts, and noise. Standard BC forces the agent to replicate the entire data distribution, including its flaws. The ideal offline RL agent should instead perform "stitching"—the ability to extract and combine the most effective segments from different trajectories to form a superior policy. This paper proposes that Optimal Transport (OT) provides the mathematical machinery necessary to achieve this selectivity.
Methodology: Partial Policy Learning (PPL)
The authors rethink the relationship between states and actions through the lens of Partial Optimal Transport (POT). In a standard OT framework, one seeks the most efficient way to transform one probability distribution into another based on a cost function. PPL adapts this to RL with several key innovations:
1. The Transport Framework
In the PPL formulation, the state distribution is treated as the source and the action distribution within the dataset as the target. The "transport cost" is defined as the negative Q-function (the critic). Under this logic, moving toward an action with a high Q-value is "cheap," while moving toward a low-value action is "expensive."
2. The "Partial" Advantage
Standard OT requires the total mass of the source to match the target, which in RL terms would mean cloning the entire dataset. PPL introduces an unbalance coefficient ($w$). This allows the algorithm to map states to only a subset of the action distribution—specifically, the subset that minimizes the transport cost. This coefficient acts as a "selectivity dial"; a higher $w$ allows the agent to be more discerning, focusing exclusively on the highest-performing actions available in the data.
3. Maximin Formulation
Unlike previous OT-based RL methods that use OT as a secondary penalty or reward, PPL views the entire policy optimization as an OT problem. By utilizing a dual maximin formulation, the authors bypass the computational hurdles typically associated with OT, such as the need for 1-Lipschitz constraints or intensive Sinkhorn iterations.
Experimental Setup
The researchers evaluated PPL on the D4RL benchmark suite, focusing on environments that test different aspects of offline learning:
- MuJoCo (Gym): Locomotion tasks (HalfCheetah, Hopper, Walker) with varying data quality (Medium, Medium-Replay, Medium-Expert).
- Antmaze: Navigation tasks that specifically require trajectory stitching to reach a goal.
- Android: High-dimensional manipulation tasks (Pen, Door, Hammer, Relocate) involving complex human and cloned data.
- Toy Experiments: A "shortest path" environment used to visualize how the algorithm ignores sub-optimal, curved paths in favor of a direct, optimal route.
Key Insights and Results
Performance Benchmarks
PPL demonstrated state-of-the-art (SOTA) performance across several categories:
- Antmaze: This was the most significant result. PPL outperformed existing methods by +16 to +21 points on the "large-play" and "large-diverse" tasks, proving its superior ability to stitch together fragmented paths.
- MuJoCo: PPL achieved an aggregate score of 799.45, slightly surpassing ReBRAC (796.9), a leading model-free baseline.
- Android: The algorithm set new records in manipulation tasks, particularly when dealing with the "Cloned" and "Human" datasets where data quality is highly inconsistent.
Implicit Stitching and Selectivity
The experiments confirmed that by minimizing the transport cost, the policy naturally gravitates toward high-value actions. In toy experiments, while standard BC produced an "average" path that was sub-optimal, PPL successfully ignored the noise to extract the true optimal trajectory. This confirms that PPL does not just clone data; it filters it.
Limitations and Future Work
While PPL offers a robust framework for offline RL, it is not without challenges:
- Hyperparameter Sensitivity: The unbalance coefficient $w$ is critical to performance but is task-dependent. Finding the optimal $w$ for new domains currently requires empirical tuning.
- Critic Reliability: Like most model-free methods, PPL's performance is bounded by the accuracy of the Q-function. If the critic provides poor value estimates, the transport cost will be inaccurate, leading to sub-optimal policies.
Future research may explore extending this framework to Hierarchical RL, using OT to map high-level goals to sub-policies, or integrating it with Weak Neural OT to better handle multimodal goal distributions.
Conclusion
Partial Policy Learning represents a significant shift in how we approach offline RL. By moving away from strict distribution matching and toward a selective transport-based optimization, PPL provides a principled way to handle the inherent imperfections of offline datasets. Its success on the Antmaze and Android benchmarks suggests that Partial Optimal Transport is a powerful tool for building agents that can learn to excel even when their training data is mediocre.
Summary updated