Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning

Yantian Zha, Lin Guan, Subbarao Kambhampati

Introduction: The Challenge of Ambiguity in RLfD

Training robots using human demonstrations is often hindered by ambiguity. For instance, if a teacher pushes a block, is the goal defined by its color, its shape, or its final location? This "causal misidentification" often leads to unstable training and poor generalization in standard Reinforcement Learning from Demonstrations (RLfD) agents.

Figure 1: The Robot-Push domain illustrating ambiguous goals and the corresponding predicate utility weights (u) learned by the agent.

The authors introduce SERLfD (Self-Explanation for RL from Demonstrations), a framework designed to bridge the gap between low-level motor skills and high-level symbolic intent. By learning to "self-explain" success through identifying critical relational features (predicates), the agent can filter out irrelevant environmental noise.

The SERLfD Framework

The core of the SERLfD approach is an interleaved process involving a Self-Explainer and a Generator.

Figure 2: Overview of the SERLfD architecture, showing the interaction between the Generator (RL agent) and the Discriminator (SE-Net).

1. The Self-Explainer (SE-Net)

The robot utilizes a vocabulary of task-agnostic predicates (e.g., block_at_blue). The SE-Net learns to assign "utility weights" ($u$) to these predicates by distinguishing between successful and failed trajectories. This contrastive learning allows the robot to identify which predicates are consistently true during success but vary during failure, effectively pinpointing the human's true intent.

2. The Generator (RL Agent)

The agent leverages the SE-Net’s outputs in two primary ways:

  • State Augmentation: Predicate values and their learned utility weights are appended to the observation vector.
  • Reward Shaping: Utility weights are used to create a potential-based reward, providing a denser signal that guides the robot toward states satisfying important predicates.

Experimental Evaluation

The framework was evaluated across several domains requiring complex manipulation and reasoning:

  • Robot-Push: A high-ambiguity task where target regions and colors are randomized.
  • Robot-Remove-and-Push: A complex 7-DoF task requiring the removal of a cover before pushing an object.
  • Pacman: A discrete domain focusing on optimal timing for eating ghosts.

The researchers used small sets of human demonstrations (8–16 trajectories) to simulate realistic low-data regimes.

Key Results and Insights

SERLfD demonstrated superior stability and performance compared to standard RLfD and GAN-IRL baselines. In the ambiguous Robot-Push domain, the agent successfully learned to prioritize color over location, allowing it to generalize even when target locations were swapped.

Ablation studies revealed that State Augmentation was particularly effective, as it provides the robot with the "why" behind its actions as part of its input. Furthermore, the use of failure data proved mathematically essential for identifying truly useful predicates versus those that are merely always true (like "being on the floor").

Conclusion and Future Work

While SERLfD significantly improves RLfD stability, it currently relies on a pre-defined vocabulary of predicates. Future work may involve integrating Large Language Models (LLMs) to automatically generate these predicates or provide initial explanations, further reducing the need for manual engineering and improving the system's ability to handle noisy real-world perception.

Summary updated

Abstract

Our work aims at efficiently leveraging ambiguous demonstrations for the training of a reinforcement learning (RL) agent. An ambiguous demonstration can usually be interpreted in multiple ways, which severely hinders the RL-Agent from learning stably and efficiently. Since an optimal demonstration may also suffer from being ambiguous, previous works that combine RL and learning from demonstration (RLfD works) may not work well. Inspired by how humans handle such situations, we propose to use self-explanation (an agent generates explanations for itself) to recognize valuable high-level relational features as an interpretation of why a successful trajectory is successful. This way, the agent can provide some guidance for its RL learning. Our main contribution is to propose the Self-Explanation for RL from Demonstrations (SERLfD) framework, which can overcome the limitations of traditional RLfD works. Our experimental results show that an RLfD model can be improved by using our SERLfD framework in terms of training stability and performance.
BibTeX
@inproceedings{zha2024learning,
  author = {Yantian Zha and Lin Guan and Subbarao Kambhampati},
  title = {{Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i9.28907},
}
Figure