Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning

Yantian Zha, Lin Guan, Subbarao Kambhampati

Introduction

Reinforcement Learning from Demonstrations (RLfD) is a cornerstone of modern robotics, allowing agents to "warm-start" their learning process using human examples. However, a significant hurdle remains: human demonstrations are inherently ambiguous. When a teacher pushes a block into a corner, is the goal the specific object, the specific location, or a combination of both? This ambiguity often leads to "causal confusion," where agents fixate on irrelevant environmental details, resulting in unstable training and poor generalization.

The SERLfD Framework

To address this, the authors introduce SERLfD (Self-Explanation for RL from Demonstrations). The core innovation is a Self-Explanation Network (SE-Net) that bridges the gap between raw sensory data and high-level symbolic reasoning.

The SERLfD architecture integrates a Self-Explanation Network (SE-Net) within an adversarial discriminator to guide policy optimization.

The framework operates through several key components:

  1. Symbolic Predicates: The robot is equipped with a library of task-agnostic logical checks (e.g., is_block_pushed, block_at_target).
  2. Utility Weighting: The SE-Net predicts utility weights ($u$) for these predicates, effectively "explaining" which relations are relevant to the current task state.
  3. Adversarial Training: By comparing successful trajectories (human or agent-generated) against failures, the SE-Net learns to highlight predicates that consistently distinguish success.
  4. Agent Guidance: These self-explanations are used for state augmentation and reward shaping, providing the RL agent with a clearer, denser signal of what truly matters.

Experimental Evaluation

The researchers tested SERLfD across diverse simulated environments, including continuous robotic manipulation (Robot-Push, Robot-Remove-and-Push) and discrete domains (Pacman). In the "Robot-Remove-and-Push" task, the agent had to navigate complex dependencies, such as removing a cover before interacting with a target block.

Performance comparison across continuous and discrete domains shows SERLfD (red/pink lines) consistently outperforming baselines in score and stability.

Key Results and Insights

The evaluation yielded three primary insights:

  • Superior Performance: SERLfD significantly outperformed vanilla RLfD baselines (like TD3fD and SACfD). In complex tasks where baselines failed to learn, SERLfD achieved high success rates.
  • Intent Disambiguation: Visual analysis of the SE-Net's output confirmed that it correctly identifies task intent. For instance, it successfully prioritized color-based goals over coordinate-based ones when the demonstrations favored the former.
  • Training Stability: Beyond raw scores, SERLfD exhibited much lower variance during training. By "explaining" the task to itself, the agent avoided the performance swings typical of standard imitation learning.

Conclusion and Future Work

SERLfD demonstrates that symbolic background knowledge can effectively guide deep reinforcement learning in ambiguous settings. While currently dependent on pre-defined predicates and reliable object tracking, the framework paves the way for more interpretable and robust robotic learning. Future work may leverage Large Language Models (LLMs) to automatically generate predicates, further reducing the need for human engineering.

Summary updated

BibTeX
@inproceedings{zha2024learning,
  author = {Yantian Zha and Lin Guan and Subbarao Kambhampati},
  title = {{Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i9.28907},
}
Figure