Introduction: The Challenge of Ambiguity in RLfD
Training robots using human demonstrations is often hindered by ambiguity. For instance, if a teacher pushes a block, is the goal defined by its color, its shape, or its final location? This "causal misidentification" often leads to unstable training and poor generalization in standard Reinforcement Learning from Demonstrations (RLfD) agents.
The authors introduce SERLfD (Self-Explanation for RL from Demonstrations), a framework designed to bridge the gap between low-level motor skills and high-level symbolic intent. By learning to "self-explain" success through identifying critical relational features (predicates), the agent can filter out irrelevant environmental noise.
The SERLfD Framework
The core of the SERLfD approach is an interleaved process involving a Self-Explainer and a Generator.
1. The Self-Explainer (SE-Net)
The robot utilizes a vocabulary of task-agnostic predicates (e.g., block_at_blue). The SE-Net learns to assign "utility weights" ($u$) to these predicates by distinguishing between successful and failed trajectories. This contrastive learning allows the robot to identify which predicates are consistently true during success but vary during failure, effectively pinpointing the human's true intent.
2. The Generator (RL Agent)
The agent leverages the SE-Net’s outputs in two primary ways:
- State Augmentation: Predicate values and their learned utility weights are appended to the observation vector.
- Reward Shaping: Utility weights are used to create a potential-based reward, providing a denser signal that guides the robot toward states satisfying important predicates.
Experimental Evaluation
The framework was evaluated across several domains requiring complex manipulation and reasoning:
- Robot-Push: A high-ambiguity task where target regions and colors are randomized.
- Robot-Remove-and-Push: A complex 7-DoF task requiring the removal of a cover before pushing an object.
- Pacman: A discrete domain focusing on optimal timing for eating ghosts.
The researchers used small sets of human demonstrations (8–16 trajectories) to simulate realistic low-data regimes.
Key Results and Insights
SERLfD demonstrated superior stability and performance compared to standard RLfD and GAN-IRL baselines. In the ambiguous Robot-Push domain, the agent successfully learned to prioritize color over location, allowing it to generalize even when target locations were swapped.
Ablation studies revealed that State Augmentation was particularly effective, as it provides the robot with the "why" behind its actions as part of its input. Furthermore, the use of failure data proved mathematically essential for identifying truly useful predicates versus those that are merely always true (like "being on the floor").
Conclusion and Future Work
While SERLfD significantly improves RLfD stability, it currently relies on a pre-defined vocabulary of predicates. Future work may involve integrating Large Language Models (LLMs) to automatically generate these predicates or provide initial explanations, further reducing the need for manual engineering and improving the system's ability to handle noisy real-world perception.
Summary updated