This paper introduces VACO (Value-aligned Behavior Cloning via Bi-level Optimization), a novel framework designed to solve the "quality vs. quantity" dilemma in offline reinforcement learning (RL). By treating policy learning as a nested optimization problem, the authors demonstrate a robust method to extract high-performance policies from sub-optimal datasets without the instability common in traditional offline RL methods.
TL;DR
VACO uses a bi-level optimization framework to automatically weight training samples in behavior cloning, prioritizing high-quality data while ignoring "noise." By aligning these weights with a value function, it achieves state-of-the-art performance across standard offline RL benchmarks like D4RL.
What Problem Does This Paper Address?
Offline RL is the art of learning a policy from a fixed dataset without any environment interaction. It faces two primary hurdles:
- Out-of-Distribution (OOD) Issues: If an agent tries an action not present in the dataset, the value function (Q-function) often overestimates its worth, leading the agent to "hallucinate" high rewards and fail.
- Value Alignment Issues: Standard Behavior Cloning (BC) is safe because it stays within the data distribution, but it is "blind"—it treats every sample in the dataset as equally good. If the dataset contains mediocre or random actions, BC will learn a mediocre policy.
As shown in the figure below, even within the "in-sample" domain, a policy might converge to a sub-optimal action ($A^-$) simply because it is more frequent in the data, missing the truly optimal action ($A^*$) suggested by value estimations.
The VACO Method
The authors propose VACO, which structures the learning process into two loops:
- The Inner Loop (Weighted BC): Instead of standard BC, VACO performs weighted behavior cloning. A "meta-scoring network" assigns an importance weight to every state-action pair in the dataset. The policy is then trained to minimize the weighted mean squared error.
- The Outer Loop (Value Alignment): This loop optimizes the parameters of the meta-scoring network. It asks: "How should the training data be weighted so that the resulting policy maximizes the expected return?"
Technically, this is implemented using a meta-scoring network (a simple 3-layer MLP) that takes the state, action, and current Q-value as input. The framework uses a "one-step differentiation" approximation—a technique borrowed from meta-learning—to calculate how changes in the scoring network affect the policy's eventual performance. This allows the system to "filter" the dataset, focusing the agent's attention on the most successful behaviors recorded in the logs.
Datasets Evaluated
The researchers evaluated VACO on several standard and challenging benchmarks:
- D4RL MuJoCo: Classic locomotion tasks (Hopper, HalfCheetah, Walker2d) with varying data quality.
- AntMaze: A navigation task that requires "trajectory stitching"—combining parts of different sub-optimal paths to find a successful route to a goal.
- Adroit & Franka Kitchen: High-dimensional robotic manipulation tasks with sparse rewards.
- V-D4RL: A vision-based version of the D4RL tasks, testing the method's ability to handle high-dimensional pixel inputs.
Key Insights
- Data Quality Filtering: Unlike methods that use complex generative models (like Diffusion or VAEs) to constrain the policy, VACO uses a simple weighting mechanism. This makes it more intuitive and easier to analyze.
- Trajectory-Level Awareness: A striking finding is that the meta-scoring network doesn't just look at individual actions; it learns to recognize high-quality trajectories. The authors found a 0.954 Pearson correlation between the average meta-weights assigned to a trajectory and that trajectory's total reward.
- Efficiency at Inference: Because the meta-scoring network is only used during training, the final policy is just a standard MLP. This ensures that VACO has the same fast inference speed as basic BC, unlike diffusion-based policies which can be computationally expensive to sample.
Key Results
VACO consistently outperformed or matched state-of-the-art baselines across almost all tasks.
- Locomotion: On the D4RL MuJoCo suite, VACO achieved an average normalized score of 95.43, surpassing strong baselines like CQL (84.20) and IQL (83.78).
- AntMaze: VACO showed superior "stitching" capabilities, particularly on the "Large" maze tasks where many other methods failed to reach the goal.
- Visual RL: In the V-D4RL benchmark, VACO maintained a significant lead over vision-based baselines, proving its robustness to different input modalities.
Limitations and Future Work
- Training Time: Bi-level optimization is computationally more demanding than single-level methods. VACO is roughly 1.7x slower to train than IQL and 4.5x slower than basic BC.
- Dependence on Q-values: While VACO helps align the policy with the value function, it still relies on a pre-trained Q-function (typically from IQL). If the initial value estimation is extremely poor, the meta-scoring network may struggle to find the right weights.
- Approximation Error: The use of "one-step differentiation" is an approximation. Future work could explore more precise (though likely more expensive) K-step differentiation to further refine the meta-weights.
Summary updated