TL;DR
This paper introduces Adaptive Probabilistic Ensemble Learning (APEL), a novel framework that allows an agent to swap between completely different reinforcement learning (RL) algorithms—even those with different state representations—on the fly. By treating base learners as "black boxes" and using a decaying Dirichlet distribution to select the best performer for each episode, APEL consistently outperforms individual algorithms and adapts to shifting environment dynamics faster than existing state-of-the-art ensemble methods.
What Problem Does This Paper Address?
Designing an RL system is often a headache of trial and error. Should you use a model-based approach (fast to learn, but prone to errors) or a model-free approach (slow but robust)? Should you use handcrafted features or raw pixels? Usually, a designer has to pick one beforehand.
While "ensemble" methods exist to combine multiple learners, most assume that every learner in the group uses the same algorithm or the same way of looking at the world (state representation). This paper tackles the challenge of heterogeneous ensembles: how do we create a "meta-learner" that can leverage a diverse toolkit of algorithms—some tabular, some deep, some model-based—and automatically figure out which one to trust at any given moment?
Methods
The authors propose APEL (Adaptive Probabilistic Ensemble Learning). Its core logic operates on two distinct timescales: the base learner timescale (individual steps within a game) and the ensemble timescale (entire episodes).
Key components include:
- Learners as Black Boxes: APEL doesn't care how a base learner works internally. It only requires that the learners can learn from "off-policy" experience (data generated by other agents).
- Dirichlet Selection: At the start of each episode, APEL samples a probability distribution from a Dirichlet distribution to decide which base learner will control the agent for that episode.
- Performance Tracking: If the chosen learner achieves a new "high score" compared to the others' recent performances, its "concentration parameter" in the Dirichlet distribution is increased, making it more likely to be picked again.
- The Decay Mechanism ($\nu$): To prevent the system from becoming "overconfident" in a single learner that might eventually plateau, APEL slightly decays all concentration parameters every episode. This creates a recency bias, allowing the agent to "forget" old successes and pivot to a different learner that is currently performing better.
Datasets
The researchers evaluated APEL across several standard and custom benchmarks:
- Maze Domain: A 10x10 randomized maze used to test training time and adaptation speed.
- OpenAI Gym Classics: CartPole-v1 and Half Cheetah-v2.
- Atari Breakout: Used specifically to test the combination of a handcrafted tabular Q-learner (simple features) and a Deep Q-Network (DQN) using raw pixels.
- Mujoco Half Cheetah: Used to test the combination of Soft Actor-Critic (SAC) and Model-Based Policy Optimization (MBPO).
Key Insights and Ideas
- Bridging the "Representation Gap": In the Breakout experiments, the handcrafted learner starts strong because its simplified state space is easy to learn, but it eventually hits a performance ceiling. The DQN starts slow but has a much higher ceiling. APEL successfully "hands off" control from the handcrafted learner to the DQN exactly when the latter becomes superior.
- Constant-Time Adaptation: Unlike the baseline method (SSBAS), which uses a sliding window that grows over time, APEL’s adaptation time remains constant. This is vital for complex tasks where an agent might train for millions of episodes; you don't want your ensemble to become "stiff" and unable to switch strategies late in the game.
- Off-Policy Synergy: Because all base learners are trained on the trajectories generated by the selected learner, the entire ensemble benefits from every episode, regardless of who was "driving."
Key Results
- Superior Average Performance: When faced with a random choice between a Maze or CartPole task, APEL achieved lower average training times than any single base learner (Q-learning, DQN, or ACER).
- Robustness to "Junk" Learners: The authors tested what happens if you pad the ensemble with many "bad" learners. They found that while performance degrades linearly as you add more bad learners, the system doesn't break—it still eventually identifies and favors the few "good" ones.
- Model-Based vs. Model-Free: In the Mujoco Half Cheetah task, APEL correctly identified MBPO as the superior learner and stuck with it, demonstrating it can handle the vastly different internal mechanics of model-based RL.
Limitations and Future Work
- Off-Policy Constraint: APEL is currently limited to off-policy algorithms. While this covers a large portion of modern RL (like DQN, SAC, and ACER), it excludes popular on-policy methods like PPO unless they are modified.
- Computational Overhead: Since APEL trains every base learner on every timestep, the computational cost scales linearly with the number of learners in the ensemble. This might be prohibitive for very large ensembles of resource-heavy deep learning models.
- Future Directions: The authors suggest exploring more complex ways to share information between learners beyond just sharing trajectories, potentially allowing for even faster convergence.
Summary updated