Abstract
Despite its promise, reinforcement learning's real-world adoption has been hampered by the need for costly exploration to learn a good policy. Imitation learning (IL) mitigates this shortcoming by using an oracle policy during training as a bootstrap to accelerate the learning process. However, in many practical situations, the learner has access to multiple suboptimal oracles, which may provide conflicting advice in a state. The existing IL literature provides a limited treatment of such scenarios. Whereas in the single-oracle case, the return of the oracle's policy provides an obvious benchmark for the learner to compete against, neither such a benchmark nor principled ways of outperforming it are known for the multi-oracle setting. In this paper, we propose the state-wise maximum of the oracle policies' values as a natural baseline to resolve conflicting advice from multiple oracles. Using a reduction of policy optimization to online learning, we introduce a novel IL algorithm MAMBA, which can provably learn a policy competitive with this benchmark. In particular, MAMBA optimizes policies by using a gradient estimator in the style of generalized advantage estimation (GAE). Our theoretical analysis shows that this design makes MAMBA robust and enables it to outperform the oracle policies by a larger margin than the IL state of the art, even in the single-oracle case. In an evaluation against standard policy gradient with GAE and AggreVaTe(D), we showcase MAMBA's ability to leverage demonstrations both from a single and from multiple weak oracles, and significantly speed up policy optimization.
Introduction
Reinforcement Learning (RL) is notoriously sample-efficient because agents must spend a vast amount of time in "trial-and-error" exploration. Imitation Learning (IL) speeds this up by providing an "oracle" (an expert policy) for the agent to mimic. However, most IL research assumes a single, near-perfect oracle.
In the real world, we often have access to multiple "experts" that are mediocre: think of a collection of simple hand-coded heuristics, different human drivers, or legacy PID controllers. These oracles often provide conflicting advice, and no single one is the best in every situation. Before this paper, there was no clear theoretical benchmark for how an agent should perform when faced with multiple suboptimal experts, nor a robust way to outperform the best of them.
The MAMBA Algorithm
The authors introduce a conceptual framework and a practical algorithm called MAMBA (Max-aggregation of Multiple Baselines).
The Max-Aggregated Baseline
The core idea is to define a new benchmark: the state-wise maximum of all oracle value functions. Instead of trying to match the average performance of experts, the agent tries to match the best possible expert for any given state. This allows the agent to "stitch together" the strengths of different experts. If Expert A is great at starting a task and Expert B is great at finishing it, MAMBA's max-aggregation naturally learns to follow A initially and switch to B later.
GAE-style Gradient Estimator
MAMBA uses a gradient estimator inspired by Generalized Advantage Estimation (GAE). It introduces a hyperparameter, $\lambda$, which acts as a "knob":
- When $\lambda=0$, the algorithm performs one-step policy improvement (classic Imitation Learning).
- When $\lambda=1$, it behaves like full Reinforcement Learning.
- By choosing a value in between (like 0.9), MAMBA can trade off the low variance of imitation with the high performance potential of RL.
This design allows MAMBA to "see beyond" the expert's immediate advice and find even better strategies, effectively using the experts as a springboard into high-performance RL.
Experimental Evaluation
The researchers evaluated MAMBA on four standard continuous control environments from the OpenAI Gym: CartPole, DoubleInvertedPendulum (DIP), HalfCheetah, and Ant. The "oracles" used were not perfect; they were partially trained neural network policies with varying degrees of suboptimality. In some experiments, they intentionally used "Bad Oracles" to test the algorithm's robustness.
Key Results
MAMBA performance across four benchmark environments (CartPole, DIP, HalfCheetah, and Ant) compared to AggreVaTeD, Policy Gradient with GAE, and the best available oracle. Source, p. 2
MAMBA demonstrated significant improvements over both pure RL and existing IL baselines:
- Significant Speedup: MAMBA learned much faster than pure RL (GAE policy gradient) by avoiding blind exploration.
- Outperforming the Best Oracle: In all environments, MAMBA eventually achieved higher returns than the best individual oracle provided to it. In HalfCheetah and Ant, the performance gain over the best oracle was substantial.
- Beating the State-of-the-Art: MAMBA outperformed AggreVaTeD (a leading single-oracle IL algorithm). In CartPole, for instance, MAMBA reached optimal performance while AggreVaTeD plateaued at a suboptimal level.
- Handling Conflict: When given 8 conflicting "bad" oracles in CartPole, MAMBA successfully filtered the noise to reach a near-perfect score, whereas single-oracle methods were highly sensitive to which specific oracle they were assigned. The diversity of demonstrations helps the agent build a better internal model of the environment's value landscape.
Conclusion and Future Work
MAMBA provides a principled way to leverage multiple suboptimal experts to accelerate reinforcement learning. While it shows great promise, there are areas for future improvement. Currently, MAMBA requires learning a value function for every oracle, which can be computationally expensive as the number of oracles grows. Future work could explore sharing neural network parameters across different oracle value functions to improve scalability. Additionally, integrating off-policy techniques could further improve sample efficiency by reusing old demonstration data more effectively.
Summary updated