Signal Temporal Logic-Guided Apprenticeship Learning

Aniruddh Gopinath Puranic, Jyotirmoy V. Deshmukh, Stefanos Nikolaidis

TL;DR

This paper introduces AL-STL, a novel framework for robot learning from demonstrations that uses Signal Temporal Logic (STL) to guide the learning of both reward functions and control policies. By leveraging a "performance graph" that captures task satisfaction and trade-offs, AL-STL significantly reduces the number of demonstrations needed to learn complex, multi-goal tasks, outperforming existing methods.

What Problem Does This Paper Address?

Learning effective control policies for robots from human demonstrations is a challenging task, especially when the desired behavior involves multiple sub-goals with temporal dependencies. Traditional methods often struggle with suboptimal or imperfect demonstrations, leading to poor reward inference and ineffective policies. Furthermore, existing approaches typically infer Markovian rewards, which fail to capture the temporal relationships between sub-goals. This paper addresses the need for a robust apprenticeship learning framework that can:

  • Learn from a limited number of suboptimal demonstrations.
  • Reason about temporal dependencies between task sub-goals.
  • Infer reward functions and control policies that are not only effective but also safe and efficient.
  • Avoid increasing the complexity of the underlying Markov Decision Process (MDP) by augmenting state or action spaces.

Methods

The core contribution of this paper is the AL-STL (Apprenticeship Learning with Signal Temporal Logic) framework. It builds upon prior work (LfD-STL) by introducing a closed-loop learning mechanism. Key components include:

  • Signal Temporal Logic (STL): Used to formally specify high-level task objectives. STL allows for quantitative evaluation of how well a demonstration or a learned policy satisfies these specifications.
  • Performance Graphs (DAGs): Temporal logic specifications are encoded into directed acyclic graphs (DAGs). These graphs capture the relationships and trade-offs between different task sub-goals as exhibited by demonstrations.
  • Performance-Graph Advantage (PGA): A novel metric derived from the performance graph. PGA quantifies the potential for improvement in a policy by considering both the overall task satisfaction (VS) and the minimization of trade-offs between specifications (ES). This metric guides the Reinforcement Learning (RL) process.
  • Closed-Loop Learning: Unlike previous open-loop methods, AL-STL iteratively updates both the reward function and the policy. Demonstrations are used to initialize the reward model, and then RL agents collect rollouts, which are evaluated using PGA. The best rollouts are maintained in a "frontier" buffer, which in turn refines the reward function for the next iteration.

Datasets

The paper evaluates the AL-STL framework on several robotic manipulation tasks using physics simulations:

  • Panda-Reach: Reaching a desired pose with the end-effector.
  • Needle-Reach: A more precise version of reaching, using a surgical robot simulation.
  • Picking and Placing: Picking up a cube and placing it at a desired location.
  • Opening Door: Opening a door by rotating and pulling.

These tasks utilize environments from PyBullet and MuJoCo, and for the surgical task, the SURROL environment. The demonstrations for these tasks are generated by training an appropriate RL agent with an expert dense reward function.

Key Insights and Ideas

  • STL as a powerful specification language: STL provides a formal and quantitative way to express complex, temporal task requirements, which is crucial for robot learning.
  • Performance graphs offer interpretable trade-offs: Encoding task specifications into DAGs provides a visual and quantitative way to understand how demonstrations satisfy different sub-goals and the trade-offs involved. This is more informative than just a single scalar reward.
  • PGA as a guiding metric for RL: The Performance-Graph Advantage metric is a key innovation. It balances maximizing overall performance with minimizing conflicting specifications, leading to more robust and well-behaved policies. This is analogous to the advantage function in RL but adapted for multi-objective temporal tasks.
  • Closed-loop learning is essential for improvement: Iteratively updating the reward function and policy based on agent exploration (rollouts) allows the framework to discover behaviors beyond the initial demonstrations, leading to better policies.
  • Sample efficiency is dramatically improved: By effectively leveraging demonstrations and guiding exploration with PGA, AL-STL requires significantly fewer demonstrations compared to state-of-the-art imitation learning (IL) and inverse reinforcement learning (IRL) methods.

Key Results

The AL-STL framework demonstrates significant improvements across various robotic manipulation tasks:

  • Reduced Demonstration Requirements: The framework achieves high success rates with as few as 5 demonstrations, whereas baseline methods often require 10x to 100x more demonstrations (e.g., 50-100 for Panda-Reach, 100 for Needle-Reach, 100 for MCAC on Door Opening).
  • High Success Rates: The method consistently achieves high success rates (e.g., over 99% for Panda-Reach and Needle-Reach, 98% for Cube-Placing, 98% for Door Opening) with minimal demonstrations.
  • Effective Policy Improvement: Experiments show that the PGA metric guides the RL agent towards better policies, with weights converging to represent uniform importance across specifications when appropriate (e.g., in the Needle-Reach task).
  • Qualitative Learning: In the Cube-Placing task, the learned policy demonstrated an ability to adapt its strategy (picking vs. pushing/dragging) based on the target location, suggesting it learned more than just mimicking demonstrations.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the performance of the algorithm is sensitive to hyperparameter settings, such as the PGA scaling factor ($\lambda$).
  • While the framework handles suboptimal demonstrations, the quality and diversity of the initial demonstrations still play a role in the learning process.
  • The current work focuses on deterministic environments for some experiments, though the framework is designed for stochastic settings.
  • Future Work:
    • Investigate the impact of demonstration diversity on learning and MDP solvability.
  • Explore probabilistic guarantees and formal verification of learned policies.
  • Address task generalization and sim-to-real transfer learning.
  • Further explore the effect of affine transformations on rewards and their impact on optimal policies.

Summary updated

Abstract

Apprenticeship learning crucially depends on effectively learning rewards, and hence control policies from user demonstrations. Of particular difficulty is the setting where the desired task consists of a number of sub-goals with temporal dependencies. The quality of inferred rewards and hence policies are typically limited by the quality of demonstrations, and poor inference of these can lead to undesirable outcomes. In this paper, we show how temporal logic specifications that describe high level task objectives, are encoded in a graph to define a temporal-based metric that reasons about behaviors of demonstrators and the learner agent to improve the quality of inferred rewards and policies. Through experiments on a diverse set of robot manipulator simulations, we show how our framework overcomes the drawbacks of prior literature by drastically improving the number of demonstrations required to learn a control policy.
BibTeX
@inproceedings{puranic2024signal,
  author = {Aniruddh Gopinath Puranic and Jyotirmoy V. Deshmukh and Stefanos Nikolaidis},
  title = {{Signal Temporal Logic-Guided Apprenticeship Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=VOptFT6rNb},
}
Figure