Value-aligned Behavior Cloning for Offline Reinforcement Learning via Bi-level Optimization

Xingyu Jiang, Ning Gao, Xiuhui Zhang, Hongkun Dou, Yue Deng

This paper introduces VACO (Value-aligned Behavior Cloning via Bi-level Optimization), a novel framework designed to solve the "quality vs. quantity" dilemma in offline reinforcement learning (RL). By treating policy learning as a nested optimization problem, the authors demonstrate a robust method to extract high-performance policies from sub-optimal datasets without the instability common in traditional offline RL methods.

TL;DR

VACO uses a bi-level optimization framework to automatically weight training samples in behavior cloning, prioritizing high-quality data while ignoring "noise." By aligning these weights with a value function, it achieves state-of-the-art performance across standard offline RL benchmarks like D4RL.

What Problem Does This Paper Address?

Offline RL is the art of learning a policy from a fixed dataset without any environment interaction. It faces two primary hurdles:

  1. Out-of-Distribution (OOD) Issues: If an agent tries an action not present in the dataset, the value function (Q-function) often overestimates its worth, leading the agent to "hallucinate" high rewards and fail.
  2. Value Alignment Issues: Standard Behavior Cloning (BC) is safe because it stays within the data distribution, but it is "blind"—it treats every sample in the dataset as equally good. If the dataset contains mediocre or random actions, BC will learn a mediocre policy.

As shown in the figure below, even within the "in-sample" domain, a policy might converge to a sub-optimal action ($A^-$) simply because it is more frequent in the data, missing the truly optimal action ($A^*$) suggested by value estimations.

Illustration of OOD and alignment issues in offline RL

The VACO Method

The authors propose VACO, which structures the learning process into two loops:

  • The Inner Loop (Weighted BC): Instead of standard BC, VACO performs weighted behavior cloning. A "meta-scoring network" assigns an importance weight to every state-action pair in the dataset. The policy is then trained to minimize the weighted mean squared error.
  • The Outer Loop (Value Alignment): This loop optimizes the parameters of the meta-scoring network. It asks: "How should the training data be weighted so that the resulting policy maximizes the expected return?"

Technically, this is implemented using a meta-scoring network (a simple 3-layer MLP) that takes the state, action, and current Q-value as input. The framework uses a "one-step differentiation" approximation—a technique borrowed from meta-learning—to calculate how changes in the scoring network affect the policy's eventual performance. This allows the system to "filter" the dataset, focusing the agent's attention on the most successful behaviors recorded in the logs.

Datasets Evaluated

The researchers evaluated VACO on several standard and challenging benchmarks:

  • D4RL MuJoCo: Classic locomotion tasks (Hopper, HalfCheetah, Walker2d) with varying data quality.
  • AntMaze: A navigation task that requires "trajectory stitching"—combining parts of different sub-optimal paths to find a successful route to a goal.
  • Adroit & Franka Kitchen: High-dimensional robotic manipulation tasks with sparse rewards.
  • V-D4RL: A vision-based version of the D4RL tasks, testing the method's ability to handle high-dimensional pixel inputs.

Key Insights

  • Data Quality Filtering: Unlike methods that use complex generative models (like Diffusion or VAEs) to constrain the policy, VACO uses a simple weighting mechanism. This makes it more intuitive and easier to analyze.
  • Trajectory-Level Awareness: A striking finding is that the meta-scoring network doesn't just look at individual actions; it learns to recognize high-quality trajectories. The authors found a 0.954 Pearson correlation between the average meta-weights assigned to a trajectory and that trajectory's total reward.
  • Efficiency at Inference: Because the meta-scoring network is only used during training, the final policy is just a standard MLP. This ensures that VACO has the same fast inference speed as basic BC, unlike diffusion-based policies which can be computationally expensive to sample.

Key Results

VACO consistently outperformed or matched state-of-the-art baselines across almost all tasks.

  • Locomotion: On the D4RL MuJoCo suite, VACO achieved an average normalized score of 95.43, surpassing strong baselines like CQL (84.20) and IQL (83.78).
  • AntMaze: VACO showed superior "stitching" capabilities, particularly on the "Large" maze tasks where many other methods failed to reach the goal.
  • Visual RL: In the V-D4RL benchmark, VACO maintained a significant lead over vision-based baselines, proving its robustness to different input modalities.
Performance comparison across various datasets

Limitations and Future Work

  • Training Time: Bi-level optimization is computationally more demanding than single-level methods. VACO is roughly 1.7x slower to train than IQL and 4.5x slower than basic BC.
  • Dependence on Q-values: While VACO helps align the policy with the value function, it still relies on a pre-trained Q-function (typically from IQL). If the initial value estimation is extremely poor, the meta-scoring network may struggle to find the right weights.
  • Approximation Error: The use of "one-step differentiation" is an approximation. Future work could explore more precise (though likely more expensive) K-step differentiation to further refine the meta-weights.

Summary updated

Abstract

Offline reinforcement learning (RL) aims to optimize policies under pre-collected data, without requiring any further interactions with the environment. Derived from imitation learning, Behavior cloning (BC) is extensively utilized in offline RL for its simplicity and effectiveness. Although BC inherently avoids out-of-distribution deviations, it lacks the ability to discern between high and low-quality data, potentially leading to sub-optimal performance when facing with poor-quality data. Current offline RL algorithms attempt to enhance BC by incorporating value estimation, yet often struggle to effectively balance these two critical components, specifically the alignment between the behavior policy and the pre-trained value estimations under in-sample offline data. To address this challenge, we propose the Value-aligned Behavior Cloning via Bi-level Optimization (VACO), a novel bi-level framework that seamlessly integrates an inner loop for weighted supervised behavior cloning (BC) with an outer loop dedicated to value alignment. In this framework, the inner loop employs a meta-scoring network to evaluate and appropriately weight each training sample, while the outer loop maximizes value estimation for alignment with controlled noise to facilitate limited exploration. This bi-level structure allows VACO to identify the optimal weighted BC policy, ultimately maximizing the expected estimated return conditioned on the learned value function. We conduct a comprehensive evaluation of VACO across a variety of continuous control benchmarks in offline RL, where it consistently achieves superior performance compared to existing state-of-the-art methods.
BibTeX
@inproceedings{jiang2025value,
  author = {Xingyu Jiang and Ning Gao and Xiuhui Zhang and Hongkun Dou and Yue Deng},
  title = {{Value-aligned Behavior Cloning for Offline Reinforcement Learning via Bi-level Optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=elTJBP7Fbv},
}
Figure