Abstract: Although reinforcement learning has found widespread use in dense reward settings, training autonomous agents with sparse rewards remains challenging. To address this difficulty, prior work has shown promising results when using not only task-specific demonstrations but also task-agnostic albeit somewhat related demonstrations. In most cases, the available demonstrations are distilled into an implicit prior, commonly represented via a single deep net. Explicit priors in the form of a database that can be queried have also been shown to lead to encouraging results. To better benefit from available demonstrations, we develop a method to Combine Explicit and Implicit Priors (CEIP). CEIP exploits multiple implicit priors in the form of normalizing flows in parallel to form a single complex prior. Moreover, CEIP uses an effective explicit retrieval and push-forward mechanism to condition the implicit priors. In three challenging environments, we find the proposed CEIP method to improve upon sophisticated state-of-the-art techniques.
Introduction
Reinforcement Learning (RL) is a powerful tool for training autonomous agents, yet it remains notoriously sample-inefficient in sparse reward environments. In these settings, a robot might only receive a signal upon the successful completion of a complex, multi-stage task. To mitigate this, researchers often leverage expert demonstrations. Traditional approaches typically distill these demonstrations into either an implicit prior (a generative model like a VAE) or an explicit prior (a retrievable database of states). However, implicit priors can lose fine-grained details, while explicit priors can be difficult to query effectively during online training. CEIP bridges this gap by combining both modalities into a unified framework.
The CEIP Framework
The CEIP framework introduces a novel pipeline that processes both task-agnostic and task-specific data to guide the RL agent.
1. Parallel Implicit Priors
Rather than training a single monolithic network, CEIP uses k-means clustering to partition task-agnostic demonstrations into distinct behavioral groups. Each group is used to train a separate Normalizing Flow. These flows provide a bijective mapping between simple and complex distributions, allowing for precise action modeling. During training, these parallel flows are combined using learned coefficients, enabling the agent to dynamically select the most relevant expert behaviors for its current state.
2. Explicit Retrieval and Push-Forward
CEIP maintains a database of task-specific demonstrations. During interaction with the environment, the agent performs a real-time lookup to identify expert states similar to its current observation. It retrieves the "likely next state" from the expert trajectory and feeds this information back into the policy. To prevent the agent from getting stuck in loops or repeating expert states, a push-forward mechanism is employed. This mechanism penalizes the retrieval of previously visited expert states, ensuring monotonic progress through the task trajectory.
Experimental Evaluation
The authors evaluated CEIP against state-of-the-art baselines, including PARROT, SKiLD, and FIST, across three challenging benchmarks: FetchReach, Kitchen (D4RL), and an Office sorting task.
Key Results
- Superior Performance: In the Kitchen-SKiLD-A benchmark, CEIP achieved a perfect success score of 4.0, significantly outperforming FIST (2.7) and SKiLD (0.06).
- Sample Efficiency: CEIP demonstrated remarkable robustness in few-shot scenarios, successfully learning complex manipulation tasks with as little as a single task-specific demonstration.
- Ablation Studies: Removing either the parallel flow architecture or the explicit retrieval mechanism resulted in a substantial performance drop, confirming that the synergy between implicit and explicit priors is critical to the method's success.
Discussion and Insights
The core innovation of CEIP lies in "horizontal stacking." By using parallel 1-layer normalizing flows instead of deep vertical layers, the model becomes more robust and easier to train. Furthermore, the explicit prior acts as a precise map, while the implicit prior provides the general behavioral "vibe," creating a complementary system that handles sparse rewards more effectively than either could alone.
Conclusion
CEIP presents a significant step forward in demonstration-guided RL. By effectively combining clustered implicit priors with a monotonic explicit retrieval system, it achieves state-of-the-art results on complex robotics tasks. Future work will explore increasing the system's robustness to sub-optimal or noisy expert demonstrations, further expanding its applicability to real-world scenarios.
Summary updated