Abstract
Composing previously mastered skills to solve novel tasks promises dramatic improvements in the data efficiency of reinforcement learning. Here, we analyze two recent works composing behaviors represented in the form of action-value functions and show that they perform poorly in some situations. As part of this analysis, we extend an important generalization of policy improvement to the maximum entropy framework and introduce an algorithm for the practical implementation of successor features in continuous action spaces. Then we propose a novel approach which addresses the failure cases of prior work and, in principle, recovers the optimal policy during transfer. This method works by explicitly learning the (discounted, future) divergence between base policies. We study this approach in the tabular case and on non-trivial continuous control problems with compositional structure and show that it outperforms or matches existing methods across all tasks considered.
Introduction
Reinforcement learning (RL) typically requires extensive interaction to master a single task. A more efficient paradigm is compositional transfer, where an agent leverages previously learned skills (e.g., "Task A" and "Task B") to solve a new, combined objective without further training. However, existing methods for zero-shot composition often falter when the optimal behavior for the combined task differs significantly from the individual base behaviors.
The Compositional Challenge: Optimism vs. Pessimism
Current approaches to policy composition generally fall into two categories:
- Generalized Policy Improvement (GPI): A "pessimistic" approach that selects the best single policy from a library. While robust, it fails to "blend" behaviors to discover novel solutions.
- Compositional Optimism (CO): Common in Maximum Entropy (Max-Ent) RL, this "optimistic" approach multiplies policies together. It assumes rewards are jointly achievable, but leads to poor performance when base policies disagree.
As shown in Figure 1, in scenarios where the target task (T) requires a specific trajectory not covered by base policies, both CO and GPI exhibit high regret. This happens because CO assumes the agent can achieve the best of both worlds simultaneously, leading to confusion when tasks are incompatible.
Divergence Correction (DC)
To address these limitations, we introduce Divergence Correction (DC). We demonstrate that the "optimism bias" in prior work is mathematically linked to the Rényi divergence between policies. By learning a "correction term" ($C^\infty$) that accounts for this divergence over time, an agent can subtract this bias from the optimistic estimate to recover the true optimal policy for any convex combination of rewards.
To implement this in continuous action spaces, we developed Adaptive Importance Sampling for Boltzmann Policies (AISBP). Since sampling from complex Boltzmann distributions is difficult, AISBP uses a mixture of Gaussians and importance sampling to estimate values and select actions efficiently.
Experimental Results
We evaluated DC against existing methods across tabular grid worlds and complex continuous control tasks (MuJoCo), including a planar manipulator and an 8D ant robot. The "tricky" versions of these tasks were specifically designed so that the optimal solution for the combined task was located in a region of the state space that neither base policy would normally visit.
In "tricky" continuous control tasks, DC significantly outperformed CO and GPI. As illustrated in Figure 6, DC successfully finds the global optimum by explicitly correcting for the divergence between base skills. While CO and GPI often get trapped in suboptimal solutions, DC's learned correction term allows it to navigate state-space regions that neither base policy would visit in isolation. Furthermore, DC matched or exceeded the performance of Conditional Q-learning (CondQ) while being easier to learn due to its decomposed structure.
Conclusion
Divergence Correction provides a principled framework for zero-shot compositional transfer. By explicitly modeling the future divergence between skills, DC bridges the gap between pessimistic selection and naive optimism, enabling agents to optimally combine behaviors in complex, high-dimensional environments.
Summary updated