Residual Q-Learning: Offline and Online Policy Customization without Value

Chenran Li, Chen Tang, Haruki Nishimura, Jean Mercat, Masayoshi Tomizuka, Wei Zhan

TL;DR

This paper introduces Residual Q-Learning (RQL), a principled framework for "policy customization"—adapting a pre-trained imitation learning policy to satisfy new downstream requirements without destroying its original imitative behavior. Crucially, RQL achieves this without needing to know the expert's underlying reward function or value function, bypassing the instabilities of Inverse Reinforcement Learning. The authors derive model-free (offline) and model-based (online) variants that successfully balance base task execution with new, often orthogonal, behavioral preferences.

What Problem Does This Paper Address?

Imitation Learning (IL) is fantastic for training agents to perform complex tasks by mimicking experts (e.g., teaching an autonomous vehicle to drive like a human). However, these learned policies are rigid; they only know how to do exactly what the demonstration showed.

If a downstream application requires a slight tweak—such as driving human-like but strictly preferring the rightmost lane, or parking while avoiding specific boundary constraints—standard Reinforcement Learning (RL) fine-tuning often fails. It either completely overwrites the imitative behavior or relies on ad-hoc regularizers (like KL-divergence constraints) that struggle when the new requirements are orthogonal to the original task.

The core challenge is: How do we mathematically blend an unknown expert reward function (implicit in the prior policy) with a known add-on reward function in a principled way?

Methods

The authors formulate policy customization as a new Markov Decision Process (MDP) where the total reward is a weighted sum of the unknown base reward ($r$) and the known add-on reward ($r_R$).

Instead of trying to learn $r$ via Inverse RL, they define a Residual Q-function: $$Q_{R,t} := \hat{Q}_t - \omega Q^*$$ where $\hat{Q}_t$ is the total action-value function and $Q^*$ is the optimal soft Q-function of the prior policy.

By substituting this into the soft Bellman equation, the unknown reward $r$ and value function $Q^*$ beautifully cancel out. The resulting update rule for $Q_R$ depends only on the known add-on reward ($r_R$) and the log-likelihood of the prior policy ($\log \pi$).

The authors leverage this math to introduce three practical algorithms:

  1. Residual Soft Q-Learning: For discrete action spaces (model-free).
  2. Residual Soft Actor-Critic (RSAC): For continuous action spaces (model-free).
  3. Residual Maximum-Entropy MCTS: A model-based approach that enables zero-shot online policy customization during planning, provided a dynamics model is available.

Datasets / Environments

The framework was evaluated across several simulated environments modified to include customized add-on objectives:

  • Classic Control: CartPole (add-on: stay centered on the rack) and Continuous Mountain Car (add-on: avoid negative/reverse actions).
  • Autonomous Driving (highway-env): Highway (add-on: stay in the rightmost lane) and Parking (add-on: avoid touching parking slot boundaries).
  • Physics Simulation (MuJoCo): Ant-v3 and Humanoid-v3 (add-on: move diagonally by adding a $y$-axis velocity reward), and Hopper-v3 (add-on: jump higher).

Key Insights and Ideas

  • Bypassing Inverse RL: Traditionally, combining unknown and known rewards required a two-step process: infer the reward using Inverse RL, then run standard RL. RQL completely bypasses this unstable bottleneck by operating directly on the prior policy's log-probabilities.
  • The Failure of Simple Regularization: Standard fine-tuning methods that simply penalize the KL-divergence from the prior policy often fail on complex tasks. Being close in probabilistic distance does not guarantee the agent will actually complete the base task. RQL succeeds because the base task's value structure is mathematically baked into the residual update.
  • Zero-Shot Adaptation: The integration with Monte Carlo Tree Search (MCTS) means an agent can instantly change its behavior profile online based on a new add-on reward without undergoing any offline gradient updates.

Key Results

Across all testbeds, RQL successfully customized policies to meet new requirements while maintaining the success rates of the base tasks:

  • Parking Environment: The baseline continuous control policy (GAIL) struggled to park accurately. When customized with RQL to avoid boundaries, the add-on reward actually acted as a guide, boosting the base task success rate from 70.91% to 83.59% while achieving a 66.07% non-violation rate (compared to just 11.27% for the uncustomized prior).
  • RSAC vs. Competitors: In the Parking task, alternative methods that augmented rewards with policy KL-divergence failed completely, resulting in a 0% success rate on the basic parking task. RQL maintained a 98.73% success rate while hitting a 96.09% non-violation rate.
  • Online Planning: In the Highway environment, the zero-shot Residual MCTS achieved performance on par with the offline, model-free RL counterparts, successfully shifting the vehicle to the rightmost lane safely.

Limitations and Future Work

  • The Prior Bottleneck: RQL is highly dependent on the quality of the prior policy. If the prior policy has poor coverage or cannot accurately represent the base task's constraints under a new trajectory distribution, the customized policy will suffer from distributional shift.
  • Maximum-Entropy Assumption: The mathematical derivation assumes the prior policy follows a maximum-entropy distribution. When tested with a standard Behavior Cloning (BC) prior (which lacks this entropy structure), RQL performed worse than standard fine-tuning.
  • Model Dependency: The online MCTS variant currently requires a perfect ground-truth dynamics model, which is unrealistic for real-world deployment. The authors note that integrating RQL with learned predictive models is a vital next step.

Summary updated

Abstract

Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. It is especially appealing for solving complex real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic human expert behavior. However, the learned imitative policy can only follow the behavior in the demonstration. When applying the imitative policy, we may need to customize the policy behavior to meet different requirements coming from diverse downstream tasks. Meanwhile, we still want the customized policy to maintain its imitative nature. To this end, we formulate a new problem setting called policy customization. It defines the learning task as training a policy that inherits the characteristics of the prior policy while satisfying some additional requirements imposed by a target downstream task. We propose a novel and principled approach to interpret and determine the trade-off between the two task objectives. Specifically, we formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines 1) the inherent reward of the demonstration; and 2) the add-on reward specified by the downstream task. We propose a novel framework, Residual Q-learning, which can solve the formulated MDP by leveraging the prior policy without knowing the inherent reward or value function of the prior policy. We derive a family of residual Q-learning algorithms that can realize offline and online policy customization, and show that the proposed algorithms can effectively accomplish policy customization tasks in various environments. Demo videos and code are available on our website: https://sites.google.com/view/residualq-learning.
BibTeX
@inproceedings{li2023residual,
  author = {Chenran Li and Chen Tang and Haruki Nishimura and Jean Mercat and Masayoshi Tomizuka and Wei Zhan},
  title = {{Residual Q-Learning: Offline and Online Policy Customization without Value}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=LaNeRwDrTk},
}
Figure