TL;DR
Teach-and-Grow Learning (TGL) is an agent-centered robotics architecture that eliminates the costly "retraining tax" of monolithic end-to-end models by abstracting few-shot demonstrations into reusable, closed-loop "Skill Blocks." Managed by a high-level AI agent, TGL dynamically composes task routes, adapts to real-time physical feedback, and stores experiences in a persistent library—allowing robots to continuously grow their capabilities without altering core model weights.
What Problem Does This Paper Address?
Current general-purpose robotics is dominated by end-to-end Vision-Language-Action (VLA) and world-action models. While elegant, these models are highly brittle when encountering out-of-distribution (OOD) scenarios, such as an unfamiliar object, a new sensor modality, or an unexpected physical contact.
When an end-to-end model fails locally, fixing it requires a global remedy: collecting new robot data, updating the entire policy, and running extensive regression testing to ensure old capabilities aren't broken. The authors term this recurring bottleneck the retraining tax. Unlike text-based AI, robot data cannot simply be scraped from the web; it must be generated through time-consuming physical operations.
TGL aims to bypass this tax by creating an architecture that can add and validate local competence incrementally without rewriting unrelated behaviors.
Methods
Instead of forcing neural network weights to absorb every nuance of the physical world, TGL introduces an agentic, modular framework that keeps foundation model weights fixed and focuses on expanding explicit skills and memory.
The architecture consists of four main pillars:
- The AI Agent Layer: A high-level orchestrator (powered by large multimodal models like GPT or Claude) that understands the task, retrieves appropriate skills, composes a working plan, and evaluates physical outcomes.
- Skill Blocks: Rather than replaying rigid trajectory fragments, a Skill Block is a closed-loop behavior defined as a contract. It encapsulates a goal, applicability scope, reusable strategy, grounding function (mapping observations to variables), compatible executors, success verifiers, and recovery choices.
- Skill Library: A persistent repository for verified, reusable Skill Blocks.
- Experience Memory: A structured textual database that records the context of successes, failures, and the specific repairs that resolved errors.
When given a new task, the robot analyzes a few successful human demonstrations to extract semantic subgoals and synthesize a shared strategy. At runtime, the agent instantiates these blocks based on the current scene geometry, executes one stage, inspects the physical consequence, and dynamically updates its route if a mismatch occurs.
Datasets
The authors evaluated TGL using the LIBERO benchmarking suite, a standard dataset designed for lifelong robot learning and knowledge transfer. Specifically, they utilized:
- LIBERO-Object: For evaluating object-centric manipulation tasks.
- LIBERO-GOAL: For analyzing feedback-driven execution and replanning traces.
Key Insights and Ideas
- Asymmetric Architecture (Reasoning vs. Control): TGL splits the cognitive load. The language-level agent handles high-level routing, tool selection, and verification, while robot-native executors (like motion planners or low-level controllers) handle metric motion and contact. This prevents a language model from acting as a direct motor authority while still allowing physical evidence to alter the plan.
- The "Teach-and-Grow" Scaling Law: The authors propose a new scaling hypothesis. Instead of scaling model parameters or raw dataset size, they scale effective reusable experience ($X$)—interaction data that has survived validation and remains retrievable. The hypothesis predicts that future-task error and the marginal teaching demand decay as power laws of $X$.
- Mutual Refinement (Jiao Xue Xiang Zhang): Inspired by classical Chinese philosophy, teaching and learning promote one another. Sparse teaching seeds exploration, and the robot's subsequent failures expose gaps, making the next round of human teaching highly targeted.
Key Results
- State-of-the-Art Performance: TGL achieves state-of-the-art success rates on the standard LIBERO benchmark.
- Decomposition and Verification: In a visual study of 10 demonstrations across condiment-transfer tasks, TGL achieved a 1.000 ordered role/type accuracy and successfully confirmed 20/20 acquisition and release effects.
- Local Growth Without Retraining: In a fixed-executor pilot study, a robot operating with a base library of 6 blocks scored 0/6 on evaluation tasks. After adding just 2 learned blocks to the library—with zero changes to model weights, runtime, or evaluation budgets—the success rate jumped to 4/6.
Limitations and Future Work
The primary trade-off in TGL is time and latency. Because the system relies on an agent loop that repeatedly observes, reasons, and calls tools sequentially, it is slower than feed-forward policy inference.
To address this, the authors propose a slow-teacher/fast-student extension as future work. In this paradigm, the slow agentic route explores frontiers and generates verified, structured interaction data. This data is then distilled into a fast student policy (like a VLA or diffusion policy) for low-latency, routine deployment, returning control to the agent only when uncertainty arises.
Summary updated