Structure from Action: Learning Interactions for Articulated Object 3D Structure Discovery

Neil Nie, Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song

Abstract

Articulated objects are abundant in daily life. Discovering their parts, joints, and kinematics is crucial for robots to interact with these objects. We introduce Structure from Action (SfA), a framework that discovers the 3D part geometry and joint parameters of unseen articulated objects via a sequence of inferred interactions. Our key insight is that 3D interaction and perception should be considered in conjunction to construct 3D articulated CAD models, especially in the case of categories not seen during training. By selecting informative interactions, SfA discovers parts and reveals initially occluded surfaces, like the inside of a closed drawer. By aggregating visual observations in 3D, SfA accurately segments multiple parts, reconstructs part geometry, and infers all joint parameters in a canonical coordinate frame. Our experiments demonstrate that a single SfA model trained in simulation can generalize to many unseen object categories with unknown kinematic structures and to real-world objects.

Introduction

For autonomous robots to function effectively in unstructured human environments, they must interact with a vast array of articulated objects, such as cabinets, appliances, and furniture. Traditional robotic perception often treats objects as rigid bodies or relies on pre-defined category templates. However, the true kinematic structure of an object—how its parts move relative to one another—is often hidden until the object is actually manipulated.

The "Structure from Action" (SfA) framework proposes a paradigm shift: instead of passive observation, the robot uses intentional interaction to "reveal" the underlying 3D structure of unfamiliar objects. By combining an intelligent interaction policy with a 3D perception module that aggregates data over time, SfA can generate complete, functional CAD models for objects it has never encountered before.

The SfA Framework

The SfA system operates in a closed loop, alternating between acting on the object and updating its internal representation. This process is driven by three core components:

1. Learned Interaction Policy

To discover structure efficiently, the robot cannot simply poke objects at random. SfA employs a Point Transformer-based network to predict informative actions. The policy identifies a "hold" point to stabilize the object's base and a "push/pull" point and direction to trigger articulation. This policy is trained using 3D scene flow as a supervisory signal, teaching the model to target regions likely to result in significant geometric changes.

2. Dynamic Part Reconstruction and 3D History

A central challenge in structure discovery is object permanence. When a robot opens a drawer, it sees the interior; when it closes it, that interior is occluded again. SfA addresses this by maintaining a volumetric "history" of the object using a 3D CNN. As the robot interacts with the object, new visual observations are aggregated into a 3D voxel grid. This allows the system to maintain a persistent belief about the object's geometry, even for parts that are currently hidden from the camera's view.

3. Joint Estimation

Once motion is induced, the system compares the object's state before and after the action. A 3D CNN analyzes these changes to infer the joint type—specifically whether it is revolute (like a door hinge) or prismatic (like a sliding drawer)—as well as the joint's axis and spatial position. This information is then used to construct a functional URDF (Unified Robot Description Format) model.

Experimental Evaluation

The SfA framework was trained and evaluated using the PartNet-Mobility dataset within the PyBullet simulation environment. To test the system's ability to generalize, it was evaluated on 77 instances from seven completely unseen categories, including lamps, folders, and glasses.

Key Results

  • Superior Accuracy: SfA demonstrated significant improvements over modular baselines (such as a combination of Ditto and UMP-Net), achieving a 25.4 percentage point increase in 3D Intersection over Union (IoU) on unseen categories.
  • Effective Interaction: The learned interaction policy proved significantly more effective than random or heuristic baselines, consistently choosing actions that revealed new parts or joints.
  • Real-World Validation: To demonstrate the robustness of the approach, the authors deployed the simulation-trained model onto a physical UR-5 robot arm. Despite the noise inherent in real-world depth sensing, SfA successfully discovered the kinematic structures of real-world objects like laptops and cabinets.

Discussion and Limitations

The primary insight of SfA is that perception and action are inseparable in the context of articulated objects. By treating interaction as a prerequisite for understanding, robots can move beyond simple recognition to functional modeling.

However, several avenues for future work remain. Currently, the framework assumes a single joint is activated per interaction, which may not hold for complex multi-jointed objects. Additionally, the system relies on a suction-based interaction model and does not yet account for physical properties like mass or friction. Future iterations could incorporate more complex grasping strategies and force-torque sensing to create even higher-fidelity digital twins of the physical world.

Conclusion

Structure from Action provides a robust, category-agnostic pathway for robots to understand the articulated world. By integrating 3D perception with active exploration, SfA enables robots to autonomously discover the functional "blueprints" of their environment, a critical step toward truly general-purpose robotic assistants.

Summary updated

Abstract

Articulated objects are abundant in daily life. Discovering their parts, joints, and kinematics is crucial for robots to interact with these objects. We introduce Structure from Action (SfA), a framework that discovers the 3D part geometry and joint parameters of unseen articulated objects via a sequence of inferred interactions. Our key insight is that 3D interaction and perception should be considered in conjunction to construct 3D articulated CAD models, especially in the case of categories not seen during training. By selecting informative interactions, SfA discovers parts and reveals initially occluded surfaces, like the inside of a closed drawer. By aggregating visual observations in 3D, SfA accurately segments multiple parts, reconstructs part geometry, and infers all joint parameters in a canonical coordinate frame. Our experiments demonstrate that a single SfA model trained in simulation can generalize to many unseen object categories with unknown kinematic structures and to real-world objects. Code and data will be publicly available.
BibTeX
@inproceedings{nie2023structure,
  author = {Neil Nie and Samir Yitzhak Gadre and Kiana Ehsani and Shuran Song},
  title = {{Structure from Action: Learning Interactions for Articulated Object 3D Structure Discovery}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342135},
}
Figure