Functional Part Decomposition for Embodied Interaction

195 concepts139 papers2 proposals

Traditional computer vision and robotic planning pipelines have long relied on semantic categorization: identifying an object as a "microwave," a "door," or a "cup." However, knowing an object's categorical label does not inherently inform an agent how to interact with it. A robot cannot grasp a category; it must grasp a specific physical surface.

To bridge this gap, a major shift has emerged toward understanding functional affordances at the part level. Instead of treating objects as monolithic entities, this paradigm decomposes environments into actionable regions defined by what an agent can do with them. By focusing on the local geometry and physical properties of functional parts (such as handles, lids, and spouts), embodied agents can drastically narrow their action search space and generalize manipulation skills to completely novel objects.

The Limits of Monolithic Labels

When an agent relies solely on semantic labels, it struggles to adapt to variations in object state or context. Knowing that an item is an "apple" does not tell a task planner whether it needs to be washed, sliced, or discarded. As Xiaowen Sun et al. demonstrate in Details Make a Difference: Object State-Sensitive Neurorobotic Task Planning, effective planning requires mapping an object's specific condition to a culturally and logically appropriate action.

Furthermore, semantic labels fail to provide the fine-grained spatial data needed for physical execution. Chengxi Ye argues in What Can I Do Around Here? Deep Functional Scene Understanding for Cognitive Robots that robotic vision must shift from asking "What is this?" to "What can I do with this?" Without this functional grounding, an agent faces an immense search space of potential actions for every new task. Nutan Chen et al. highlight that identifying affordances serves as a crucial cognitive heuristic, narrowing down valid interactions in complex 3D scenes to accelerate learning and execution.

[Traditional Pipeline]  Visual Input ──> Semantic Label ("Drawer") ──> Intractable Action Search
[Affordance Pipeline]   Visual Input ──> Functional Part ("Handle") ──> Direct Motion Primitive ("Pull")

Why Part-Centric Abstraction Generalizes

The core intuition behind functional part decomposition is that global object geometries vary wildly, but local functional geometries remain highly consistent. A refrigerator door, a microwave, and a filing cabinet look entirely different as whole objects. However, their handles all share a localized, graspable structure that supports a pulling or pushing motion primitive.

The framework introduced in Adaptive Articulated Object Manipulation On The Fly with Foundation Model Reasoning and Part Grounding leverages this insight by decomposing complex manipulation tasks into the consistent local geometry of individual parts. This allows the system to achieve zero-shot generalization on novel objects. Similarly, Pengwei Xie et al. show in Part-Guided 3D RL for Sim2Real Articulated Object Manipulation that representing articulated objects as collections of functional parts allows a single reinforcement learning policy to manipulate diverse items like doors and drawers without needing category-specific training.

By mapping diverse geometric structures to shared functional labels, models can transfer skills seamlessly. As shown by Yue Chen et al. in PA3FF:Learning Part-Aware Dense 3D Feature Field For Generalizable Articulated Object Manipulation, focusing on functional parts rather than specific instances allows a robot to successfully interact with unseen components like handles or lids across entirely different object categories.

Formalizing the Affordance Triad

To translate visual perception into motor control, affordances are structured as a relational triad. As reviewed by Paola Ardón et al. in Building Affordance Relations for Robotic Agents - A Review, this triad connects a target object/part, an action, and the resulting effect.

       [Action Primitive] (e.g., Twist)
             /        \
            /          \
           v            v
[Functional Part] ──> [Resulting Effect]
 (e.g., Bottle Cap)     (e.g., Opened)

This structured representation bridges high-level semantic understanding and low-level physical interaction. In GMAP: Generalized Manipulation of Articulated Objects in Robotic Using Pre-trained Model, Hongliang Zeng utilizes affordance mapping to identify potential interaction points and assign success scores to action proposals, highlighting exactly where a robot should apply force.

This spatial mapping is often extended across entire environments as an "affordance landscape." Tushar Nagarajan et al. demonstrate in Learning Affordance Landscapes for Interaction Exploration in 3D Environments that creating a visual map of interactable regions accelerates multi-step downstream tasks like cooking or cleaning. When tasks require complex coordination, Björn S. Plonka shows in Learning Spatial Bimanual Action Models Based on Affordance Regions and Human Demonstrations that modeling tasks as spatial relationships between specific functional regions (such as a container's spout and a cup's rim) allows bimanual robots to generalize skills regardless of global object shapes.

Grounding Foundation Models in Physical Mechanics

While modern Vision-Language Models (VLMs) excel at abstract semantic associations, they frequently fail to understand the physical prerequisites of manipulation. Atharva Gundawar et al. expose this limitation in PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?, showing that generalist models often recognize only the most obvious use case for an object, leading to rigid behavior.

To achieve true embodied intelligence, models must integrate direct spatial and structural inputs. For example, ShapeLLM, introduced by Zekun Qi et al. in ShapeLLM: Universal 3D Object Understanding for Embodied Interaction, pairs language models with 3D point clouds to force an understanding of the mechanics of object parts and their functional purposes.

When a model successfully grounds language in physical affordances, it unlocks creative problem-solving capabilities, such as tool substitution. As Lakshmi Nair et al. argue in Creative Problem Solving in Large Language and Vision Models - What Would it Take?, prompting models with the specific physical properties required for a task (such as weight, shape, or material) allows them to bridge the conceptual gap between a missing standard tool and an unconventional replacement—enabling an agent to realize that a heavy saucepan can function perfectly well as a substitute hammer.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_object_affordance_understanding_4de34fd2,
  title = {object affordance understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-affordance-understanding/rdr_4de34fd25e3b4dc1a180b65cfded2089},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_object_affordance_understanding_4de34fd2_v1,
  title = {object affordance understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-affordance-understanding/rdr_4de34fd25e3b4dc1a180b65cfded2089/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake