TL;DR
This paper introduces Residual Q-Learning (RQL), a principled framework for "policy customization"—adapting a pre-trained imitation learning policy to satisfy new downstream requirements without destroying its original imitative behavior. Crucially, RQL achieves this without needing to know the expert's underlying reward function or value function, bypassing the instabilities of Inverse Reinforcement Learning. The authors derive model-free (offline) and model-based (online) variants that successfully balance base task execution with new, often orthogonal, behavioral preferences.
What Problem Does This Paper Address?
Imitation Learning (IL) is fantastic for training agents to perform complex tasks by mimicking experts (e.g., teaching an autonomous vehicle to drive like a human). However, these learned policies are rigid; they only know how to do exactly what the demonstration showed.
If a downstream application requires a slight tweak—such as driving human-like but strictly preferring the rightmost lane, or parking while avoiding specific boundary constraints—standard Reinforcement Learning (RL) fine-tuning often fails. It either completely overwrites the imitative behavior or relies on ad-hoc regularizers (like KL-divergence constraints) that struggle when the new requirements are orthogonal to the original task.
The core challenge is: How do we mathematically blend an unknown expert reward function (implicit in the prior policy) with a known add-on reward function in a principled way?
Methods
The authors formulate policy customization as a new Markov Decision Process (MDP) where the total reward is a weighted sum of the unknown base reward ($r$) and the known add-on reward ($r_R$).
Instead of trying to learn $r$ via Inverse RL, they define a Residual Q-function: $$Q_{R,t} := \hat{Q}_t - \omega Q^*$$ where $\hat{Q}_t$ is the total action-value function and $Q^*$ is the optimal soft Q-function of the prior policy.
By substituting this into the soft Bellman equation, the unknown reward $r$ and value function $Q^*$ beautifully cancel out. The resulting update rule for $Q_R$ depends only on the known add-on reward ($r_R$) and the log-likelihood of the prior policy ($\log \pi$).
The authors leverage this math to introduce three practical algorithms:
- Residual Soft Q-Learning: For discrete action spaces (model-free).
- Residual Soft Actor-Critic (RSAC): For continuous action spaces (model-free).
- Residual Maximum-Entropy MCTS: A model-based approach that enables zero-shot online policy customization during planning, provided a dynamics model is available.
Datasets / Environments
The framework was evaluated across several simulated environments modified to include customized add-on objectives:
- Classic Control: CartPole (add-on: stay centered on the rack) and Continuous Mountain Car (add-on: avoid negative/reverse actions).
- Autonomous Driving (
highway-env): Highway (add-on: stay in the rightmost lane) and Parking (add-on: avoid touching parking slot boundaries). - Physics Simulation (MuJoCo): Ant-v3 and Humanoid-v3 (add-on: move diagonally by adding a $y$-axis velocity reward), and Hopper-v3 (add-on: jump higher).
Key Insights and Ideas
- Bypassing Inverse RL: Traditionally, combining unknown and known rewards required a two-step process: infer the reward using Inverse RL, then run standard RL. RQL completely bypasses this unstable bottleneck by operating directly on the prior policy's log-probabilities.
- The Failure of Simple Regularization: Standard fine-tuning methods that simply penalize the KL-divergence from the prior policy often fail on complex tasks. Being close in probabilistic distance does not guarantee the agent will actually complete the base task. RQL succeeds because the base task's value structure is mathematically baked into the residual update.
- Zero-Shot Adaptation: The integration with Monte Carlo Tree Search (MCTS) means an agent can instantly change its behavior profile online based on a new add-on reward without undergoing any offline gradient updates.
Key Results
Across all testbeds, RQL successfully customized policies to meet new requirements while maintaining the success rates of the base tasks:
- Parking Environment: The baseline continuous control policy (GAIL) struggled to park accurately. When customized with RQL to avoid boundaries, the add-on reward actually acted as a guide, boosting the base task success rate from 70.91% to 83.59% while achieving a 66.07% non-violation rate (compared to just 11.27% for the uncustomized prior).
- RSAC vs. Competitors: In the Parking task, alternative methods that augmented rewards with policy KL-divergence failed completely, resulting in a 0% success rate on the basic parking task. RQL maintained a 98.73% success rate while hitting a 96.09% non-violation rate.
- Online Planning: In the Highway environment, the zero-shot Residual MCTS achieved performance on par with the offline, model-free RL counterparts, successfully shifting the vehicle to the rightmost lane safely.
Limitations and Future Work
- The Prior Bottleneck: RQL is highly dependent on the quality of the prior policy. If the prior policy has poor coverage or cannot accurately represent the base task's constraints under a new trajectory distribution, the customized policy will suffer from distributional shift.
- Maximum-Entropy Assumption: The mathematical derivation assumes the prior policy follows a maximum-entropy distribution. When tested with a standard Behavior Cloning (BC) prior (which lacks this entropy structure), RQL performed worse than standard fine-tuning.
- Model Dependency: The online MCTS variant currently requires a perfect ground-truth dynamics model, which is unrealistic for real-world deployment. The authors note that integrating RQL with learned predictive models is a vital next step.
Summary updated