Abstract
Composing previously mastered skills to solve novel tasks promises dramatic improvements in the data efficiency of reinforcement learning. Here, we analyze two recent works composing behaviors represented in the form of action-value functions and show that they perform poorly in some situations. As part of this analysis, we extend an important generalization of policy improvement to the maximum entropy framework and introduce an algorithm for the practical implementation of successor features in continuous action spaces. Then we propose a novel approach which addresses the failure cases of prior work and, in principle, recovers the optimal policy during transfer. This method works by explicitly learning the (discounted, future) divergence between base policies. We study this approach in the tabular case and on non-trivial continuous control problems with compositional structure and show that it outperforms or matches existing methods across all tasks considered.
Introduction
Reinforcement Learning (RL) has achieved remarkable success in complex domains, yet it remains notoriously data-inefficient, typically requiring agents to learn every new task from scratch. In contrast, biological intelligence excels at transfer learning—the ability to combine previously learned skills (e.g., walking and juggling) to solve novel, complex tasks with minimal additional practice.
This paper addresses the challenge of "zero-shot" skill composition. When a new task's reward can be expressed as a combination of rewards from known tasks, how can an agent synthesize an optimal policy without further environment interaction? Prior approaches generally fall into two categories: Generalized Policy Improvement (GPI), which is often too pessimistic by only selecting the single best existing policy, and Compositional Optimism (CO), which simply sums action-value functions ($Q$-functions) and often becomes overly optimistic when skills demand contradictory actions.
Divergence Correction (DC)
The authors introduce Divergence Correction (DC) to bridge the gap between these pessimistic and optimistic bounds. The core insight is that the failure of simple skill blending (CO) is a direct result of policy divergence. If two skills require vastly different actions, the value of attempting to combine them is lower than the sum of their individual parts.
Mathematically, the authors prove that the optimism bias in CO is exactly related to the discounted sum of Rényi divergences between the base policies. By introducing a correction term, $C^\infty$, learned via a Temporal Difference (TD) estimator, the agent can explicitly account for this "cost of disagreement" over time. This allows for the synthesis of an approximately optimal policy for a new reward combination instantly.
To implement this in continuous action spaces, the authors developed Adaptive Importance Sampling for Boltzmann Policies (AISBP). This algorithm allows the agent to sample from a composed action-value function online by using a mixture of existing policy proposals and importance sampling.
Experimental Results
The proposed framework was evaluated across a range of environments, from tabular grid worlds to high-dimensional continuous control tasks in MuJoCo and the DeepMind Control Suite, including a planar manipulator, a jumping ball, and an 8-DOF Ant.
Tabular Analysis
In an 8x8 grid world, the limitations of prior methods are starkly visible. GPI often fails to find the optimal path when it requires a blend of skills, while CO's over-optimism leads to high regret. DC consistently recovers the optimal policy, matching the performance of an agent trained from scratch on the specific task combination.
Continuous Control
In more complex tasks, such as the 8-DOF Ant reaching a specific target region, DC outperformed both Max-Ent GPI and Compositional Optimism. In scenarios where the optimal path did not resemble any single base skill, DC was the only method capable of consistently finding the solution. The authors also introduced "DC-Cheap," a heuristic version using a Gaussian assumption that performed surprisingly well, suggesting the approach is robust for practical applications.
Conclusion and Future Work
Divergence Correction provides a principled way to achieve zero-shot transfer by explicitly modeling the future divergence between policies. While highly effective, the authors note that scaling to very large libraries of skills (hundreds or more) remains a challenge for learning the correction term. Additionally, the current framework assumes shared environment dynamics across tasks. Future research may explore extending these compositional techniques to non-stationary dynamics or even larger-scale skill hierarchies.
Summary updated