Reconfiguring Procedural Knowledge for Compositional Robot Skill Adaptation

Daehee Lee, Dongsu Lee, Sanghyun Ahn, Minjong Yoo, Honguk WooarXiv 2026 · arXiv

TL;DR

RecPo (Reconfigurable Procedural knowledge) is a new framework that allows robots to adapt to complex, long-horizon tasks by restructuring their existing skills rather than retraining from scratch. By organizing procedural knowledge into "multi-view" graphs, RecPo can resolve vague task instructions into precise execution sequences in just 1.0 ms, achieving 91.9% success on challenging benchmarks.

What Problem Does This Paper Address?

Modern robots are increasingly powered by massive Vision-Language-Action (VLA) models that internalize a wealth of procedural knowledge. However, deploying these models in the real world remains a challenge. A robot trained in a lab might struggle with a slightly different kitchen layout or a new set of user instructions.

Current adaptation methods generally fall into two camps:

  1. Full Retraining: Costly, slow, and prone to "catastrophic forgetting" (where the robot loses old skills while learning new ones).
  2. Behavior Retrieval: The robot looks for a similar past experience in its memory. While efficient, these methods often fail to understand how to compose individual skills into a coherent sequence, especially when the task instructions are "sparse" (e.g., providing only a few snapshots of the desired goal rather than a step-by-step guide).

The authors introduce the concept of reconfigurability: the ability to realize new behaviors by cost-efficiently restructuring existing components and relations. The goal is to enable "on-site" adaptation using limited computational resources.

Methods

RecPo tackles this through an "update-and-compose" architecture consisting of two main components:

1. Procedure Knowledge Base (PKB)

Instead of storing experiences as a single, monolithic memory, RecPo breaks them down into multiple procedure views. These views are object-centric (e.g., focusing only on the microwave or the kettle).

  • Skill-Transition Graphs: For each view, RecPo builds a graph where nodes represent states and edges represent skills that cause transitions. This preserves local connectivity—knowing which skill can follow another.
  • Salience Marks: It calculates which view is most "salient" for a given skill. For example, the "fill-level" view is highly salient for a pouring skill, while the "background" view is not.

2. Task Materialization

When a new task arrives as a sequence of sparse waypoints, RecPo:

  • Abstracts the Query: It projects the waypoints into the same multi-view space used in the PKB.
  • Resolves the Path: It searches the stored graphs to find a sequence of skills that matches the desired state changes across all relevant views.
  • Grounds the Execution: The resulting "task quotient" is a sequence of executable skills tailored to the specific environment.
Figure 1: Conceptual overview of reconfigurable skill policy under sparse governing information. (left) Sparse task evidence specifies desired changes, while procedure-side evidence provides task- agnostic skill records observed through multiple procedure views. (middle) Behavior retrieval accesses stored records independently, whereas structured procedural approaches expose connectivity among reusable behaviors. (right) RecPo converts observed view-level changes into a compositional query and materializes a task-specific executable skill sequence over view-consistent procedural transitions.

Datasets

The authors evaluated RecPo using:

  • Extended Franka Kitchen: A benchmark based on the standard Franka Kitchen environment but extended to include long-horizon tasks (up to 7 subtasks) that were never seen during training. This forces the model to compose skills in entirely new ways.
  • Real-Robot Study: A physical Franka Emika Research 3 robot was used for a "pour coffee" task. The study tested the robot's ability to adapt to different cup sizes and background environments through local memory edits.

Key Insights and Ideas

  • Factorization Over Embeddings: Traditional methods often collapse all information into a single embedding space. RecPo’s "multi-view" approach prevents "aliasing"—where two different states look the same to the robot because it's looking at the wrong features (e.g., focusing on the background instead of the water level in a cup).
  • Structural Constraints Matter: The paper shows that simply scaling up the search space for skill sequences (combinatorial search) yields diminishing returns. By using view-consistent graphs, RecPo constrains the search to transitions that are physically grounded and compatible.
  • Efficiency as a First-Class Citizen: By treating adaptation as a memory operation (adding or editing records in a graph) rather than a gradient-descent problem, RecPo achieves inference speeds (1.0 ms) that allow for near-instantaneous re-planning.
RecPo organizes records into view-local graphs to expose connectivity.

Key Results

  • High Success Rate: RecPo achieved 91.9% ordered completion on the extended Franka Kitchen benchmark, significantly outperforming retrieval-based (8.1%) and skill-chaining (51.8%) baselines.
  • Robustness to Sparsity: Even when the task demonstration had large gaps between waypoints (high sparsity), RecPo maintained high performance (89.3%–95.4%), whereas retrieval methods collapsed.
  • Real-World Adaptation: In real-robot tests, RecPo successfully handled scenarios requiring a "stopping" condition (to prevent overfilling a cup) by simply adding a new procedure record and a "fill-level" view to its memory. It achieved 10/10 sequence success, while the Skill Chaining baseline failed completely (0/10).

Limitations and Future Work

  • View Dependency: RecPo currently relies on a supplied set of views (e.g., object-centric factors). While the authors show these can be derived from image embeddings (using DINOv3), the framework's performance is inherently tied to the quality of these views.
  • State-Only Records: Currently, skill records only store start and end states. This is sufficient for many tasks, but some skills might require more granular trajectory information to be successfully composed.
  • Causal Independence: The framework assumes subtasks are somewhat causally independent. Future work could explore how to handle tasks with more complex, interlocking physical dependencies.

RecPo represents a significant step toward robots that can truly learn on the job, turning the problem of "learning a new task" into a much simpler problem of "rearranging what you already know."

Summary updated

Abstract

General-purpose behavior models increasingly internalize broad procedural knowledge during training, making their adaptation to new goals and skill interfaces less a problem of learning entirely new behaviors than of reconfiguring existing competence. We study this capability as reconfigurability, the ability to realize new behaviors by cost-efficiently restructuring existing components and relations under sparse governing information, where task specifications provide high-level goals and procedure specifications expose a limited set of reusable factors, leaving the concrete bindings and compositions required for execution underdetermined. We propose RecPo, a framework that represents skills task-agnostically through multiple procedure views and materializes executable skill sequences by composing stored transitions to realize task-specific changes. RecPo maintains a procedure knowledge base of skill-induced transitions, with each view exposing a reusable factor and preserving local skill connectivity. At deployment, it interprets task specifications through the same views and resolves the resulting compositional queries over the stored relations to bind skills for execution. On long-horizon tasks in an extended Franka Kitchen benchmark, RecPo achieves 91.9% ordered completion with 1.0 ms of task inference per episode across varying procedure support and task-record sparsity. Real-robot studies further demonstrate adaptation through local memory edits without task-specific retraining.
BibTeX
@misc{lee2026reconfiguring,
  author = {Daehee Lee and Dongsu Lee and Sanghyun Ahn and Minjong Yoo and Honguk Woo},
  title = {{Reconfiguring Procedural Knowledge for Compositional Robot Skill Adaptation}},
  year = {2026},
  eprint = {2611.22174},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22174},
}
Figure