TL;DR
Reinforcement learning (RL) is often slow because agents must explore environments by trial and error. While imitation learning (IL) speeds this up using an "expert" oracle, real-world experts are often suboptimal or disagree with each other. This paper introduces MAMBA, an algorithm that stitches together advice from multiple weak oracles to create a policy that outperforms them all, significantly accelerating the path to optimal performance.
The Challenge of Multiple Flawed Experts
RL has a massive "exploration problem": to find a good strategy, an agent usually has to stumble around a complex environment for a long time. IL solves this by giving the agent an "oracle" (an expert) to follow.
However, most IL research assumes access to a single, near-perfect expert. In reality, you might have several "okay" experts—perhaps a human driver who is great at city traffic but poor on highways, and a cruise control system that is the opposite. Standard IL struggles here: how do you resolve conflicting advice? How do you know which expert to trust in a specific state? Before this paper, there wasn't even a clear theoretical benchmark for what "success" looked like when learning from multiple, flawed experts.
Introducing MAMBA
The authors propose a new framework and a practical algorithm called MAMBA (Max-aggregation of Multiple Baselines) to address these challenges.
- The Max-Aggregated Baseline: The core theoretical insight is that the agent should aim for the "state-wise maximum" of all available oracles. If Oracle A has a higher expected value in state $X$, but Oracle B is better in state $Y$, the agent's target should be the best of both at every single step.
- Reduction to Online Learning: MAMBA treats policy optimization as a sequence of decisions where the "adversary" is the environment's dynamics.
- GAE-style Gradient Estimator: MAMBA uses a gradient estimator inspired by Generalized Advantage Estimation (GAE). It introduces a hyperparameter, $\lambda$, which acts as a "knob" to trade off between pure imitation (fast but potentially biased) and pure RL (slow but unbiased).
- RIRO (Roll-In/Roll-Out): The agent interacts with the environment by "rolling in" with its current policy and then letting an oracle "roll out" to see the long-term consequences of its actions.
Key Results and Performance
The researchers evaluated MAMBA on four continuous control environments from OpenAI Gym (powered by DART and MuJoCo): CartPole (8 oracles), DoubleInvertedPendulum or DIP (4 oracles), Halfcheetah (2 oracles), and Ant (2 oracles). Crucially, the "oracles" used were not perfect; they were partially trained neural networks with varying degrees of competence.
- Speed and Performance: In all environments, MAMBA reached higher returns much faster than standard RL (GAE policy gradient) and single-oracle IL (AggreVaTeD).
- Outperforming the Best: Even when the single-oracle baseline was given the best available expert, MAMBA (using all experts) still learned faster and reached a higher final performance.
- Bootstrap Capability: In the "Ant" environment, MAMBA demonstrated a dramatic speedup, reaching high performance levels in under 100 iterations, while standard RL barely made progress in the same timeframe.
Insights and Future Directions
MAMBA proves that you don't need a perfect expert to get a perfect agent. By stitching together the best behaviors of several "weak" oracles, MAMBA forms a baseline stronger than any individual contributor. Furthermore, its focus on the state-wise maximum makes it naturally robust to "noisy" or conflicting advice.
Limitations: MAMBA requires learning a value function for every oracle. As the number of oracles grows, the computational cost and the amount of data needed to approximate these values increase. In higher-dimensional problems, the benefit of adding more oracles was smaller, likely because it became harder to accurately learn all the different oracle value functions simultaneously.
Future Work: The authors suggest that future work could use "parameter sharing" (using one large neural network to represent all oracle values) or off-policy learning to make the algorithm more efficient as the number of experts scales up.
Summary updated