Policy Improvement via Imitation of Multiple Oracles

Ching-An Cheng, Andrey Kolobov, Alekh Agarwal

TL;DR

Reinforcement learning (RL) is often slow because agents must explore environments by trial and error. While imitation learning (IL) speeds this up using an "expert" oracle, real-world experts are often suboptimal or disagree with each other. This paper introduces MAMBA, an algorithm that stitches together advice from multiple weak oracles to create a policy that outperforms them all, significantly accelerating the path to optimal performance.

The Challenge of Multiple Flawed Experts

RL has a massive "exploration problem": to find a good strategy, an agent usually has to stumble around a complex environment for a long time. IL solves this by giving the agent an "oracle" (an expert) to follow.

However, most IL research assumes access to a single, near-perfect expert. In reality, you might have several "okay" experts—perhaps a human driver who is great at city traffic but poor on highways, and a cruise control system that is the opposite. Standard IL struggles here: how do you resolve conflicting advice? How do you know which expert to trust in a specific state? Before this paper, there wasn't even a clear theoretical benchmark for what "success" looked like when learning from multiple, flawed experts.

Introducing MAMBA

The authors propose a new framework and a practical algorithm called MAMBA (Max-aggregation of Multiple Baselines) to address these challenges.

  1. The Max-Aggregated Baseline: The core theoretical insight is that the agent should aim for the "state-wise maximum" of all available oracles. If Oracle A has a higher expected value in state $X$, but Oracle B is better in state $Y$, the agent's target should be the best of both at every single step.
  2. Reduction to Online Learning: MAMBA treats policy optimization as a sequence of decisions where the "adversary" is the environment's dynamics.
  3. GAE-style Gradient Estimator: MAMBA uses a gradient estimator inspired by Generalized Advantage Estimation (GAE). It introduces a hyperparameter, $\lambda$, which acts as a "knob" to trade off between pure imitation (fast but potentially biased) and pure RL (slow but unbiased).
  4. RIRO (Roll-In/Roll-Out): The agent interacts with the environment by "rolling in" with its current policy and then letting an oracle "roll out" to see the long-term consequences of its actions.

Key Results and Performance

The researchers evaluated MAMBA on four continuous control environments from OpenAI Gym (powered by DART and MuJoCo): CartPole (8 oracles), DoubleInvertedPendulum or DIP (4 oracles), Halfcheetah (2 oracles), and Ant (2 oracles). Crucially, the "oracles" used were not perfect; they were partially trained neural networks with varying degrees of competence.

Performance across four environments
  • Speed and Performance: In all environments, MAMBA reached higher returns much faster than standard RL (GAE policy gradient) and single-oracle IL (AggreVaTeD).
  • Outperforming the Best: Even when the single-oracle baseline was given the best available expert, MAMBA (using all experts) still learned faster and reached a higher final performance.
  • Bootstrap Capability: In the "Ant" environment, MAMBA demonstrated a dramatic speedup, reaching high performance levels in under 100 iterations, while standard RL barely made progress in the same timeframe.

Insights and Future Directions

MAMBA proves that you don't need a perfect expert to get a perfect agent. By stitching together the best behaviors of several "weak" oracles, MAMBA forms a baseline stronger than any individual contributor. Furthermore, its focus on the state-wise maximum makes it naturally robust to "noisy" or conflicting advice.

Limitations: MAMBA requires learning a value function for every oracle. As the number of oracles grows, the computational cost and the amount of data needed to approximate these values increase. In higher-dimensional problems, the benefit of adding more oracles was smaller, likely because it became harder to accurately learn all the different oracle value functions simultaneously.

Future Work: The authors suggest that future work could use "parameter sharing" (using one large neural network to represent all oracle values) or off-policy learning to make the algorithm more efficient as the number of experts scales up.

Summary updated

Abstract

Despite its promise, reinforcement learning’s real-world adoption has been hampered by the need for costly exploration to learn a good policy. Imitation learning (IL) mitigates this shortcoming by using an oracle policy during training as a bootstrap to accelerate the learning process. However, in many practical situations, the learner has access to multiple suboptimal oracles, which may provide conflicting advice in a state. The existing IL literature provides a limited treatment of such scenarios. Whereas in the single-oracle case, the return of the oracle’s policy provides an obvious benchmark for the learner to compete against, neither such a benchmark nor principled ways of outperforming it are known for the multi-oracle setting. In this paper, we propose the state-wise maximum of the oracle policies’ values as a natural baseline to resolve conflicting advice from multiple oracles. Using a reduction of policy optimization to online learning, we introduce a novel IL algorithm MAMBA, which can provably learn a policy competitive with this benchmark. In particular, MAMBA optimizes policies by using a gradient estimator in the style of generalized advantage estimation (GAE). Our theoretical analysis shows that this design makes MAMBA robust and enables it to outperform the oracle policies by a larger margin than the IL state of the art, even in the single-oracle case. In an evaluation against standard policy gradient with GAE and AggreVaTe(D), we showcase MAMBA’s ability to leverage demonstrations both from a single and from multiple weak oracles, and significantly speed up policy optimization.
BibTeX
@inproceedings{cheng2020policy,
  author = {Ching-An Cheng and Andrey Kolobov and Alekh Agarwal},
  title = {{Policy Improvement via Imitation of Multiple Oracles}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=Ute2dQ3452C},
}
Figure