In reinforcement learning and robotics, a single policy often struggles to balance competing demands. An agent might need to reach a goal quickly while simultaneously avoiding obstacles, maintaining safety constraints, and adapting to unfamiliar environments. Traditionally, researchers have addressed this by training a single, monolithic network or by summing different control vectors. However, these approaches often lead to "brittle" agents that fail under distribution shifts or produce unsafe "compromise" actions when objectives conflict.
A growing body of research proposes a different path: policy fusion. Instead of relying on one model, these methods synthesize a global controller from an ensemble of specialized experts, teachers, or prior behaviors. By treating policies as probability distributions and combining them through Bayesian or multiplicative frameworks, agents can leverage the strengths of multiple sub-optimal sources to outperform any single one.
The Limits of Additive Blending
The simplest way to combine two policies is through a weighted average or convex combination. For instance, Li et al. propose the "I Know How" (IKH) framework, where a "Master Policy" learns to weight the contributions of pre-trained expert skills. Similarly, Shi et al. combine standard reinforcement learning (RL) with distributional RL to balance average-case performance with risk-aware robustness.
While effective for simple skill-stitching, additive methods act like a logical OR. If one expert suggests moving left to avoid a wall and another suggests moving right to reach a goal, an additive agent might average these into a disastrous decision to move straight into the wall. As Liu et al. argue, vector summation often leads to local minima where opposing forces cancel each other out.
Multiplicative Composition as Logical Intersection
To solve the "compromise" problem, recent work has shifted toward multiplicative policy composition. In this view, each policy is treated as a stochastic energy distribution. When you multiply these distributions together, the resulting policy acts as a probabilistic logical AND.
Anqi Li et al. demonstrate this through Composable Energy Policies (CEP). By multiplying independent energy-based experts, the agent only assigns high probability to actions that are acceptable to every module. This ensures that safety constraints are never "averaged away" by the desire to reach a goal. Yu and Li apply a similar principle to quadruped locomotion, using the product of Gaussian distributions to switch between agile gaits without the jerky transitions typical of discrete switching.
Bayesian Fusion: Prior Knowledge and Likelihood
Policy fusion also provides a mathematically rigorous way to integrate prior knowledge with new experiences. In this context, an existing rule-based controller or a human expert acts as a prior, while the agent's current learning provides a likelihood.
Malato and Hautamaki use this Bayesian approach to enhance imitation learning. By treating a learned neural policy as a prior and updating it with a likelihood derived from retrieved expert demonstrations, the agent maintains generalization while staying grounded in reliable historical data. Han and Yilmaz apply this to autonomous driving, allowing a rule-based expert to dominate early in training (when the RL agent is uncertain) and gradually handing over control as the agent's confidence increases.
State-Specific Expertise and Selection
Not all experts are useful in all situations. A key challenge in fusion is determining who to trust and when. Beliaev et al. introduced ILEED, a framework that estimates the expertise of different demonstrators as a function of the state. This allows the agent to "filter" the noise of sub-optimal teachers and only learn from the best behaviors for specific sub-tasks.
This concept of state-dependent selection is central to several recent architectures:
- Active Selection: Liu et al. developed MAPS, which "interviews" multiple sub-optimal oracles and uses uncertainty-based exploration to identify the best expert for a given region of the state space.
- Hierarchical Management: Subramanian proposes a two-level actor-critic where a high-level "manager" learns to switch between specialized "worker" experts who may only understand parts of a complex task.
- Spatial Partitioning: Mankowitz et al. show that optimizing the boundaries between sub-policies (where one expert hands off to another) is just as important as optimizing the policies themselves.
Why Fusion Works
The power of policy fusion lies in its ability to synthesize a "super-expert" from a committee of flawed ones. As shown by Guohao Li et al. in the OIL framework, an agent can exceed the performance of its individual teachers by evaluating the relative advantage of their actions. By aggregating specialized knowledge—whether from human feedback Cederborg, physical models Xu et al., or diverse RL update rules Pengdeng Li et al.—fusion transforms the problem of "learning from scratch" into a problem of intelligent aggregation.
Key Foundational Work
Active Learning in Partially Observable Markov Decision Processes (2005) — contributed 1 concept elements to this direction.
Flexible Decision-Making in Sequential Auctions (2004) — contributed 1 concept elements to this direction.