Abstract
In Learning from Demonstrations (LfD), extracting the underlying intent and priority of tasks from human behavior is a core challenge. This paper introduces PeGLearn, an algorithm that automatically infers a "performance graph"—a Directed Acyclic Graph (DAG) representing the hierarchy of task priorities—directly from demonstrations. By utilizing Signal Temporal Logic (STL) to evaluate rule satisfaction, PeGLearn can derive robust reward functions from imperfect data, eliminating the need for manual expert ranking. Our results across multiple simulated and real-world environments show that PeGLearn achieves high success rates with significantly fewer demonstrations than existing methods.
Introduction
A primary goal in robotics is to enable agents to learn complex tasks by observing human demonstrations. However, human demonstrators are often sub-optimal, making mistakes or choosing inefficient paths. To guide the learning process, researchers use Signal Temporal Logic (STL) to define formal task specifications. While the LfD-STL framework is powerful, it requires a pre-defined performance graph to rank these specifications (e.g., "safety" must outweigh "speed"). Manually constructing these graphs becomes prohibitively difficult as the number of rules increases. PeGLearn addresses this bottleneck by automating the discovery of these priority hierarchies from raw demonstration data.
The PeGLearn Methodology
PeGLearn follows a structured pipeline to transform raw demonstrations into a functional performance graph used for Reinforcement Learning.
The algorithm operates in three primary stages:
- Local Graph Generation: For each demonstration, the system calculates "robustness" scores for every STL specification. A local DAG is created where edges represent a clear preference of one rule over another based on satisfaction levels.
- Aggregation: These local graphs are merged into a single "interim graph" by averaging edge weights across all demonstrations. This step captures the general intent but may introduce cycles due to variations in demonstrator behavior.
- DAG Conversion: To ensure the graph is compatible with Reinforcement Learning agents, cycles are systematically broken. The algorithm compares bidirectional edges and retains only the strongest net preferences, resulting in a final DAG.
Once the final DAG is established, weights are assigned to each node based on its hierarchical position, which then shapes the reward function for the agent.
Experimental Evaluation
The effectiveness of PeGLearn was validated across four diverse environments: 2D Autonomous Driving, MiR100 Navigation, the CARLA Driving Simulator, and the JIGSAWS surgical robot dataset.
Key Results
- High Success Rates: In the MiR100 task, PeGLearn achieved a 90% success rate, nearly matching the 93% performance of a manually-tuned expert reward function.
- Sample Efficiency: The method required at least 4x fewer demonstrations than traditional Inverse Reinforcement Learning (IRL) approaches to achieve similar performance levels.
- Robustness to Noise: The algorithm successfully learned correct policies even when 50% of the demonstrations were failures, highlighting its ability to filter intent from sub-optimal data.
Discussion and Insights
PeGLearn offers several advantages over traditional search-based ranking methods. By building the graph bottom-up from data, it achieves a computational complexity of $O(mn^2)$, making it scalable to complex tasks with many rules. Furthermore, a user study with 146 participants confirmed that PeGLearn's inferred rankings align closely with human expert intuition, outperforming standard clustering and random baselines.
One limitation is the "feasibility assumption," which requires at least one demonstration to satisfy all specifications to some degree. Future work will explore "deep temporal learning" to predict priority labels for new demonstrations without re-running the full graph-building process.
Conclusion
PeGLearn provides a robust, automated solution for discovering task hierarchies in LfD. By bridging the gap between raw, imperfect demonstrations and formal logic specifications, it enables robots to learn complex behaviors more efficiently and with less human intervention.
Summary updated