CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations

Kai Yan, Alex Schwing, Yu-Xiong Wang

Reinforcement Learning (RL) has achieved incredible feats, from mastering board games to controlling complex robots. However, it still hits a major wall in "sparse reward" environments—scenarios where an agent only receives feedback after completing a long sequence of actions. Without a constant stream of "good job" signals, agents often spend millions of steps flailing aimlessly.

The paper CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations introduces a novel way to bridge this gap by combining two different methods of learning from human demonstrations.

TL;DR

CEIP (Combining Explicit and Implicit Priors) significantly improves the sample efficiency of reinforcement learning by simultaneously using "implicit" learned action distributions and "explicit" state-retrieval from a demonstration database. By blending these two types of guidance, the agent can navigate complex, sparse-reward robotic tasks far more effectively than previous state-of-the-art methods.

The Challenge of Sparse Rewards

The core challenge addressed by CEIP is sample inefficiency in sparse-reward RL. When rewards are rare, exploration is difficult. To solve this, researchers often use "priors"—knowledge extracted from demonstrations. Existing methods usually fall into two camps:

  1. Implicit Priors: These distill demonstrations into a neural network (like a VAE or a Normalizing Flow) that predicts likely actions. While flexible, they can "wash out" specific, crucial details from the original data.
  2. Explicit Priors: These maintain a database of actual demonstration trajectories. At each step, the agent looks up a similar state in the database to see what to do next. While precise, these can be brittle; if the agent drifts slightly off-track from the demo, the "lookup" might fail or get the agent stuck in a loop.

The authors argue that we shouldn't have to choose. CEIP is designed to get the best of both worlds.

Methods: The CEIP Framework

Overview of the CEIP architecture.

The CEIP framework consists of three main stages:

1. The Implicit Prior: Parallel Normalizing Flows

Instead of training one giant model on all demonstrations, CEIP uses k-means clustering to group "task-agnostic" demonstrations (data from related but different tasks) into similar skills. It then trains multiple independent, 1-layer Normalizing Flows in parallel.

To adapt to a specific target task, CEIP learns a set of coefficients to "mix" these parallel flows into one large, complex prior. This allows the model to selectively use the most relevant parts of the task-agnostic data.

2. The Explicit Prior: Push-Forward Retrieval

At every step of RL training, the agent queries a small database of task-specific demonstrations. It finds the state most similar to its current one and retrieves the next state from that demonstration ($\hat{s}_{next}$).

To solve the "getting stuck" problem common in explicit priors, the authors introduce a push-forward mechanism. This adds a penalty to the retrieval process that discourages the agent from revisiting the same state or moving backward in a trajectory, effectively forcing it to keep making progress.

3. Conditioning

The final action is sampled from the combined Normalizing Flow, which is conditioned on both the current state $s$ and the retrieved "likely next state" $\hat{s}_{next}$ from the explicit prior.

Key Insights

  • Hybridization is Key: The "aha moment" is that implicit and explicit priors are complementary. The implicit prior provides a smooth "action map," while the explicit prior provides a concrete "waypoint" ($\hat{s}_{next}$) to aim for.
  • Normalizing Flows over VAEs: The authors chose Normalizing Flows because they are bijective (invertible). This ensures the agent maintains control over the entire action space, preventing the "mode collapse" or restricted exploration sometimes seen in VAE-based skill models.
  • The Power of the Push-Forward: Without the push-forward penalty in the database lookup, the explicit prior often fails because the agent gets stuck in a loop, repeatedly retrieving the same "next state" without ever actually reaching it.

Key Results

The authors evaluated CEIP on three increasingly difficult robotic manipulation environments: FetchReach (a 3D robot arm reaching task), Kitchen (a 7-DOF robot arm performing sequential tasks), and Office (a highly complex sorting environment).

CEIP outperforms baseline methods across complex tasks.

CEIP consistently outperformed sophisticated baselines like PARROT, SKiLD, and FIST across all benchmarks:

  • FetchReach: CEIP achieved near-perfect rewards very quickly, while baselines either took much longer or failed to reach the goal entirely.
  • Kitchen: In all tested scenarios, CEIP maintained a higher average reward and solved tasks more reliably than the competition.
  • Office: In this high-dimensional, long-horizon task, the explicit prior proved essential. CEIP was the only method that showed significant, steady improvement in reward, whereas variants without the explicit prior or the push-forward mechanism flatlined.

Limitations and Future Work

While powerful, CEIP has a few limitations. Training dozens of parallel flows can be time-consuming, though this can be mitigated by parallelizing the training process. Additionally, the method assumes the demonstrations are relatively optimal; poor demos might lead the agent astray. Finally, finding the "sweet spot" for the number of parallel flows (degrees of freedom) versus the risk of overfitting remains an area for future research.

Summary updated

Abstract

Although reinforcement learning has found widespread use in dense reward settings, training autonomous agents with sparse rewards remains challenging. To address this difficulty, prior work has shown promising results when using not only task-specific demonstrations but also task-agnostic albeit somewhat related demonstrations. In most cases, the available demonstrations are distilled into an implicit prior, commonly represented via a single deep net. Explicit priors in the form of a database that can be queried have also been shown to lead to encouraging results. To better benefit from available demonstrations, we develop a method to Combine Explicit and Implicit Priors (CEIP). CEIP exploits multiple implicit priors in the form of normalizing flows in parallel to form a single complex prior. Moreover, CEIP uses an effective explicit retrieval and push-forward mechanism to condition the implicit priors. In three challenging environments, we find the proposed CEIP method to improve upon sophisticated state-of-the-art techniques.
BibTeX
@inproceedings{yan2022ceip,
  author = {Kai Yan and Alex Schwing and Yu-Xiong Wang},
  title = {{CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=R5KjUket6w},
}
Figure