Reinforcement Learning (RL) has achieved incredible feats, from mastering board games to controlling complex robots. However, it still hits a major wall in "sparse reward" environments—scenarios where an agent only receives feedback after completing a long sequence of actions. Without a constant stream of "good job" signals, agents often spend millions of steps flailing aimlessly.
The paper CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations introduces a novel way to bridge this gap by combining two different methods of learning from human demonstrations.
TL;DR
CEIP (Combining Explicit and Implicit Priors) significantly improves the sample efficiency of reinforcement learning by simultaneously using "implicit" learned action distributions and "explicit" state-retrieval from a demonstration database. By blending these two types of guidance, the agent can navigate complex, sparse-reward robotic tasks far more effectively than previous state-of-the-art methods.
The Challenge of Sparse Rewards
The core challenge addressed by CEIP is sample inefficiency in sparse-reward RL. When rewards are rare, exploration is difficult. To solve this, researchers often use "priors"—knowledge extracted from demonstrations. Existing methods usually fall into two camps:
- Implicit Priors: These distill demonstrations into a neural network (like a VAE or a Normalizing Flow) that predicts likely actions. While flexible, they can "wash out" specific, crucial details from the original data.
- Explicit Priors: These maintain a database of actual demonstration trajectories. At each step, the agent looks up a similar state in the database to see what to do next. While precise, these can be brittle; if the agent drifts slightly off-track from the demo, the "lookup" might fail or get the agent stuck in a loop.
The authors argue that we shouldn't have to choose. CEIP is designed to get the best of both worlds.
Methods: The CEIP Framework
The CEIP framework consists of three main stages:
1. The Implicit Prior: Parallel Normalizing Flows
Instead of training one giant model on all demonstrations, CEIP uses k-means clustering to group "task-agnostic" demonstrations (data from related but different tasks) into similar skills. It then trains multiple independent, 1-layer Normalizing Flows in parallel.
To adapt to a specific target task, CEIP learns a set of coefficients to "mix" these parallel flows into one large, complex prior. This allows the model to selectively use the most relevant parts of the task-agnostic data.
2. The Explicit Prior: Push-Forward Retrieval
At every step of RL training, the agent queries a small database of task-specific demonstrations. It finds the state most similar to its current one and retrieves the next state from that demonstration ($\hat{s}_{next}$).
To solve the "getting stuck" problem common in explicit priors, the authors introduce a push-forward mechanism. This adds a penalty to the retrieval process that discourages the agent from revisiting the same state or moving backward in a trajectory, effectively forcing it to keep making progress.
3. Conditioning
The final action is sampled from the combined Normalizing Flow, which is conditioned on both the current state $s$ and the retrieved "likely next state" $\hat{s}_{next}$ from the explicit prior.
Key Insights
- Hybridization is Key: The "aha moment" is that implicit and explicit priors are complementary. The implicit prior provides a smooth "action map," while the explicit prior provides a concrete "waypoint" ($\hat{s}_{next}$) to aim for.
- Normalizing Flows over VAEs: The authors chose Normalizing Flows because they are bijective (invertible). This ensures the agent maintains control over the entire action space, preventing the "mode collapse" or restricted exploration sometimes seen in VAE-based skill models.
- The Power of the Push-Forward: Without the push-forward penalty in the database lookup, the explicit prior often fails because the agent gets stuck in a loop, repeatedly retrieving the same "next state" without ever actually reaching it.
Key Results
The authors evaluated CEIP on three increasingly difficult robotic manipulation environments: FetchReach (a 3D robot arm reaching task), Kitchen (a 7-DOF robot arm performing sequential tasks), and Office (a highly complex sorting environment).
CEIP consistently outperformed sophisticated baselines like PARROT, SKiLD, and FIST across all benchmarks:
- FetchReach: CEIP achieved near-perfect rewards very quickly, while baselines either took much longer or failed to reach the goal entirely.
- Kitchen: In all tested scenarios, CEIP maintained a higher average reward and solved tasks more reliably than the competition.
- Office: In this high-dimensional, long-horizon task, the explicit prior proved essential. CEIP was the only method that showed significant, steady improvement in reward, whereas variants without the explicit prior or the push-forward mechanism flatlined.
Limitations and Future Work
While powerful, CEIP has a few limitations. Training dozens of parallel flows can be time-consuming, though this can be mitigated by parallelizing the training process. Additionally, the method assumes the demonstrations are relatively optimal; poor demos might lead the agent astray. Finally, finding the "sweet spot" for the number of parallel flows (degrees of freedom) versus the risk of overfitting remains an area for future research.
Summary updated