TL;DR
RecPo (Reconfigurable Procedural knowledge) is a new framework that allows robots to adapt to complex, long-horizon tasks by restructuring their existing skills rather than retraining from scratch. By organizing procedural knowledge into "multi-view" graphs, RecPo can resolve vague task instructions into precise execution sequences in just 1.0 ms, achieving 91.9% success on challenging benchmarks.
What Problem Does This Paper Address?
Modern robots are increasingly powered by massive Vision-Language-Action (VLA) models that internalize a wealth of procedural knowledge. However, deploying these models in the real world remains a challenge. A robot trained in a lab might struggle with a slightly different kitchen layout or a new set of user instructions.
Current adaptation methods generally fall into two camps:
- Full Retraining: Costly, slow, and prone to "catastrophic forgetting" (where the robot loses old skills while learning new ones).
- Behavior Retrieval: The robot looks for a similar past experience in its memory. While efficient, these methods often fail to understand how to compose individual skills into a coherent sequence, especially when the task instructions are "sparse" (e.g., providing only a few snapshots of the desired goal rather than a step-by-step guide).
The authors introduce the concept of reconfigurability: the ability to realize new behaviors by cost-efficiently restructuring existing components and relations. The goal is to enable "on-site" adaptation using limited computational resources.
Methods
RecPo tackles this through an "update-and-compose" architecture consisting of two main components:
1. Procedure Knowledge Base (PKB)
Instead of storing experiences as a single, monolithic memory, RecPo breaks them down into multiple procedure views. These views are object-centric (e.g., focusing only on the microwave or the kettle).
- Skill-Transition Graphs: For each view, RecPo builds a graph where nodes represent states and edges represent skills that cause transitions. This preserves local connectivity—knowing which skill can follow another.
- Salience Marks: It calculates which view is most "salient" for a given skill. For example, the "fill-level" view is highly salient for a pouring skill, while the "background" view is not.
2. Task Materialization
When a new task arrives as a sequence of sparse waypoints, RecPo:
- Abstracts the Query: It projects the waypoints into the same multi-view space used in the PKB.
- Resolves the Path: It searches the stored graphs to find a sequence of skills that matches the desired state changes across all relevant views.
- Grounds the Execution: The resulting "task quotient" is a sequence of executable skills tailored to the specific environment.
Datasets
The authors evaluated RecPo using:
- Extended Franka Kitchen: A benchmark based on the standard Franka Kitchen environment but extended to include long-horizon tasks (up to 7 subtasks) that were never seen during training. This forces the model to compose skills in entirely new ways.
- Real-Robot Study: A physical Franka Emika Research 3 robot was used for a "pour coffee" task. The study tested the robot's ability to adapt to different cup sizes and background environments through local memory edits.
Key Insights and Ideas
- Factorization Over Embeddings: Traditional methods often collapse all information into a single embedding space. RecPo’s "multi-view" approach prevents "aliasing"—where two different states look the same to the robot because it's looking at the wrong features (e.g., focusing on the background instead of the water level in a cup).
- Structural Constraints Matter: The paper shows that simply scaling up the search space for skill sequences (combinatorial search) yields diminishing returns. By using view-consistent graphs, RecPo constrains the search to transitions that are physically grounded and compatible.
- Efficiency as a First-Class Citizen: By treating adaptation as a memory operation (adding or editing records in a graph) rather than a gradient-descent problem, RecPo achieves inference speeds (1.0 ms) that allow for near-instantaneous re-planning.
Key Results
- High Success Rate: RecPo achieved 91.9% ordered completion on the extended Franka Kitchen benchmark, significantly outperforming retrieval-based (8.1%) and skill-chaining (51.8%) baselines.
- Robustness to Sparsity: Even when the task demonstration had large gaps between waypoints (high sparsity), RecPo maintained high performance (89.3%–95.4%), whereas retrieval methods collapsed.
- Real-World Adaptation: In real-robot tests, RecPo successfully handled scenarios requiring a "stopping" condition (to prevent overfilling a cup) by simply adding a new procedure record and a "fill-level" view to its memory. It achieved 10/10 sequence success, while the Skill Chaining baseline failed completely (0/10).
Limitations and Future Work
- View Dependency: RecPo currently relies on a supplied set of views (e.g., object-centric factors). While the authors show these can be derived from image embeddings (using DINOv3), the framework's performance is inherently tied to the quality of these views.
- State-Only Records: Currently, skill records only store start and end states. This is sufficient for many tasks, but some skills might require more granular trajectory information to be successfully composed.
- Causal Independence: The framework assumes subtasks are somewhat causally independent. Future work could explore how to handle tasks with more complex, interlocking physical dependencies.
RecPo represents a significant step toward robots that can truly learn on the job, turning the problem of "learning a new task" into a much simpler problem of "rearranging what you already know."
Summary updated