Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models

Hamidreza Kasaei, Georgios Tziafas, Hamidreza Kasaei 0001

TL;DR

This paper introduces LRLL, a novel lifelong learning agent for robots that continuously expands its skill library using Large Language Models (LLMs). LRLL addresses the limitations of static skill sets and prompt engineering in current LLM-based robotics by enabling robots to learn new, composable, and generalizable skills over time, even from non-expert human guidance.

What Problem Does This Paper Address?

Robotics has seen a surge in using Large Language Models (LLMs) to generate robot control policies. However, existing approaches often rely on pre-defined, static libraries of skills and require extensive, expert-crafted prompt engineering. This limits the robot's ability to adapt to new tasks and learn continuously, leading to a "stationary" range of capabilities. Furthermore, traditional methods often suffer from catastrophic forgetting when trying to learn new skills, making lifelong learning a significant challenge. The paper aims to overcome these limitations by creating a robot that can autonomously grow its own library of skills, becoming more capable over time without constant human intervention or complex retraining.

Methods

The core of LRLL is a "wake-sleep" learning cycle inspired by biological learning.

  • Wake Phase: The agent interacts with its environment (simulated or real) to gather new experiences. An LLM-based exploration module, guided by human demonstrations and hints, proposes new tasks. An actor-critic LLM then generates policies and success verification code for these proposed tasks. Successful task-policy pairs are stored in an "experience memory."
  • Sleep Phase: The agent "reflects" on its experiences. Experiences are clustered based on their code structure. An LLM then distills these clusters into new, abstract skills (Python functions) that are added to the robot's "skill library." These new skills are then used to "refactor" the existing experiences, effectively compressing the memory and preparing for the next cycle.

Key novelties include:

  • Soft Memory Module: Dynamically stores and retrieves past experiences for context.
  • Self-Guided Exploration Policy: Proposes new tasks in simulation.
  • Skill Abstractor: Distills experiences into new library skills.
  • Lifelong Learning Algorithm: Enables non-expert users to bootstrap new skills with minimal interaction, bypassing gradient-based optimization to avoid catastrophic forgetting.

Datasets

The paper uses a simulated tabletop environment built on the PyBullet simulator and the Ravens manipulation suite. They introduce a curriculum of 4 cycles: Spatial Coordination, Visual Reasoning, Object Manipulation, and Rearrangement. For evaluation, they generate task instances in three splits: seen instructions with seen attributes (SA), seen instructions with unseen attributes (UA), and unseen instructions with unseen attributes (UI). They also define forward-transfer (FT) and backward-transfer (BT) splits to evaluate lifelong learning capabilities.

Key Insights and Ideas

  • Lifelong Learning via Skill Abstraction: The "sleep" phase, where experiences are distilled into new skills, is crucial for enabling lifelong learning and avoiding catastrophic forgetting. This allows the robot to build a more robust and generalizable skill set over time.
  • Exploration is Key for Generalization: The self-guided exploration policy actively seeks out new task variations and compositions, significantly expanding the agent's experience beyond initial demonstrations and leading to better generalization.
  • LLMs as Skill Builders, Not Just Policy Generators: LRLL leverages LLMs not just to execute tasks but to actively create and refine the fundamental building blocks (skills) of its own capabilities.
  • Memory Compression and Efficiency: The sleep phase effectively compresses the experience memory, reducing the number of experiences needed to maintain performance and leading to more efficient prompt retrieval.
  • Sim-to-Real Transferability: The approach demonstrates successful transfer to a real dual-arm robot setup without further adaptation, highlighting the practical applicability of the learned skills.

Key Results

  • LRLL outperforms baselines like CLIPort and LLM-static in generalization to unseen attributes and instructions, showing an average increase of ~6% in unseen attributes and ~10% in unseen instructions compared to LLM-static.
  • The exploration module (wake phase) significantly improves performance, with LRLL-no-wake showing consistently lower results.
  • The sleep phase (abstraction) leads to a substantial reduction in the number of experiences required to maintain performance, with an 8x decrease in RAM usage for storing experience embeddings.
  • LRLL demonstrates strong forward transfer capabilities, with a ~15% delta over static baselines when retrieving explored experiences.
  • The system successfully transferred learned skills to a real dual-arm robot for complex rearrangement tasks.

Limitations and Future Work

  • Perception Limitations: Current perception is limited to referring expressions and attribute classification. Future work aims to incorporate multimodal LLMs for more open-ended vision-language grounding.
  • Demonstration Input: Human demonstration input is currently language-based. The authors suggest augmenting this with video or kinesthetic teaching for more complex, contact-rich manipulation tasks.
  • Domain/Engine Adaptation: Initial prompts for exploration and abstraction modules may need refinement when changing domains or LLM engines.
  • LLM Latency and Cost: Exploration with commercial LLMs is constrained by latency and price. Future work may investigate open-source alternatives.

Summary updated

Abstract

Large Language Models (LLMs) have emerged as a new paradigm for embodied reasoning and control, most recently by generating robot policy code that utilizes a custom library of vision and control primitive skills. However, prior arts fix their skills library and steer the LLM with carefully hand-crafted prompt engineering, limiting the agent to a stationary range of addressable tasks. In this work, we introduce LRLL, an LLM-based lifelong learning agent that continuously grows the robot skill library to tackle manipulation tasks of ever-growing complexity. LRLL achieves this with four novel contributions: 1) a soft memory module that allows dynamic storage and retrieval of past experiences to serve as context, 2) a self-guided exploration policy that proposes new tasks in simulation, 3) a skill abstractor that distills recent experiences into new library skills, and 4) a lifelong learning algorithm for enabling human users to bootstrap new skills with minimal online interaction. LRLL continuously transfers knowledge from the memory to the library, building composable, general and interpretable policies, while bypassing gradient-based optimization, thus relieving the learner from catastrophic forgetting. Empirical evaluation in a simulated tabletop environment shows that LRLL outperforms end-to-end and vanilla LLM approaches in the lifelong setup while learning skills that are transferable to the real world. Project material will become available at the webpage https://gtziafas.github.io/LRLL_project.
BibTeX
@inproceedings{kasaei2024lifelong,
  author = {Hamidreza Kasaei and Georgios Tziafas and Hamidreza Kasaei},
  title = {{Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10611448},
}
Figure