Skill Transfer for Temporally-Extended Task Specifications

Jason Xinyu Liu, Ankit Shah, Eric Rosen, George Konidaris, Stefanie Tellex

Introduction

Reinforcement Learning (RL) has demonstrated significant potential in enabling robots to perform complex behaviors. However, a major hurdle remains: RL is notoriously data-intensive. This challenge is amplified when tasks are specified using Linear Temporal Logic (LTL), a formal language capable of defining intricate sequences and safety constraints (e.g., "Pick up the mail, then enter the house, but never step on the grass"). Traditionally, RL agents treat each new LTL formula as an entirely new problem, failing to reuse knowledge from similar previous tasks. This lack of compositionality prevents robots from scaling to diverse, real-world missions without prohibitive amounts of retraining.

Methodology: The LTL-Transfer Algorithm

To address this, we introduce LTL-Transfer, a zero-shot algorithm that enables robots to solve novel, complex tasks by recombining "skills" learned from prior missions. The approach functions in two primary phases:

1. Transition-Centric Option Compilation

Building on the hierarchical RL "Options Framework," we decompose tasks into a Reward Machine (RM). An RM is a finite state machine where edges represent logical propositions. Instead of learning a single policy for an entire formula, we "repackage" learned policies into transition-centric options. Each option is specifically tied to an edge in the RM, representing a skill that moves the environment from one logical state to another.

2. Zero-Shot Composition and Relaxed Matching

When presented with a novel LTL task, the system generates the corresponding RM and searches for a path through it that can be executed using the existing library of transition-centric options. A critical innovation here is the Relaxed Edge-Matching Condition. Using formal methods, the algorithm determines if a previously learned skill is "safe enough" for a new requirement. It ensures the skill's behavior is a logical subset of the new task's needs and guarantees that no "unrecoverable failure states" (safety violations) will be triggered.

Overview of LTL-Transfer in a gridworld environment, including the reward machine graph for a specific task formula.

Experimental Evaluation

The effectiveness of LTL-Transfer was evaluated in two distinct environments:

  • Minecraft-inspired Gridworld: A simulated 19x19 environment containing various objects like wood, grass, and tools. The agent was trained on 50 tasks and tested on 100 "challenging" unseen tasks and 300 "common" tasks.
  • Physical Robot Demonstration: The system was deployed on a Boston Dynamics Spot quadruped. After training on 20 navigation tasks in a lab, the robot was tested on 100 novel "fetch-and-deliver" and navigation missions.
A Boston Dynamics Spot robot performing a navigation and manipulation task in a laboratory setting.

Key Results and Insights

The experiments yielded several significant findings:

  • High Success and Zero-Shot Efficiency: In the gridworld domain, LTL-Transfer solved over 90% of challenging unseen tasks and 100% of simpler novel tasks without any additional environment interaction or training.
  • Guaranteed Safety: Leveraging the formal properties of LTL, the algorithm achieved zero safety violations across all tests. If a safe skill sequence could not be found, the robot simply would not proceed, avoiding the "guessing" behavior common in neural-network-based transfer methods.
  • Robustness: The system remained resilient in stochastic environments, with success rates dropping by only ~3% when noise (e.g., slipping) was introduced.

Limitations and Future Directions

While promising, the current framework focuses on co-safe LTL, which applies to tasks completed in finite time. Future work will aim to extend this to "infinite-horizon" tasks, such as continuous patrolling. Additionally, while the "Relaxed" matching condition is powerful, the system's success is still bound by the diversity of its initial skill library. We also envision incorporating "intra-option" updates, allowing the robot to refine and optimize transferred skills during execution to improve efficiency.

Summary updated

BibTeX
@inproceedings{liu2024skill,
  author = {Jason Xinyu Liu and Ankit Shah and Eric Rosen and George Konidaris and Stefanie Tellex},
  title = {{Skill Transfer for Temporally-Extended Task Specifications}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=vI4TXdVOP7},
}
Figure