Abstract
Understanding the mechanical structure of articulated objects is a fundamental requirement for robotic interaction. Current approaches often rely on extensive manual labeling or high-fidelity 3D sensors, which limit their scalability and real-world applicability. This paper presents SM³ (Self-Supervised Multi-task Modeling), a framework that reconstructs textured 3D models and estimates articulation structures using only multi-view RGB images. By leveraging differentiable rendering and a novel PatchImage Loss, SM³ identifies movable parts and rotation axes by comparing an object's state before and after interaction. Our results demonstrate that SM³ significantly outperforms existing state-of-the-art methods in both joint estimation accuracy and reconstruction quality.
Introduction
For a robot to effectively manipulate objects in a human environment—such as opening a dishwasher or closing a laptop—it must understand the object's "articulation structure." This involves identifying which parts are movable and determining the precise location and orientation of the joints that connect them. Traditionally, this has been a difficult computer vision task, often requiring either massive datasets with manual joint annotations or expensive depth sensors that provide high-fidelity 3D point clouds.
SM³ addresses these limitations by proposing a self-supervised system that learns from standard 2D images. By observing an object in two different states (e.g., open and closed), the system infers the underlying mechanical constraints without requiring pre-existing 3D templates or category-specific labels.
The Challenge of Articulated Object Modeling
Existing solutions for articulated object modeling typically face two major hurdles:
- Heavy Supervision: Many models are trained on category-specific data. While they may understand how a "laptop" functions, they often fail when encountering novel objects like a specialized toolbox because they lack generalizable physical priors.
- Sensor Requirements: Methods that rely on 3D point clouds are sensitive to noise and incompleteness, which are common in real-world settings where lighting and surface reflectivity vary.
SM³ bridges this gap by utilizing differentiable rendering to bridge the 2D-3D divide, allowing the system to optimize 3D parameters directly from 2D image observations.
Methodology
The SM³ pipeline is built on the principle of differentiable rendering, allowing the model to backpropagate errors from 2D image mismatches to 3D joint parameters. The process follows a multi-step approach:
1. Tetrahedral Reconstruction
The system utilizes the Nvdiffrec framework to construct a 3D model of the object using a deformable tetrahedral grid. This reconstruction is performed for both the "before" and "after" interaction states.
2. Movable Part Segmentation Prior
By comparing the 3D models of the two states, the system identifies which vertices have changed position. This comparison generates a "segmentation prior," which serves as an initial map distinguishing the movable part (e.g., a door) from the static base.
3. Candidate Joint Prediction
To determine the rotation axis, SM³ applies Principal Component Analysis (PCA) to the static base to establish a local coordinate system. It then matches parallel edges of bounding boxes between the static and movable components to identify the most probable pivot point and axis of rotation.
4. Integrated Optimization and PatchImage Loss
This is the core innovation of the framework. The system simulates the rotation of the movable part around the predicted joint and compares this virtual rendering to the actual "after-interaction" images. To refine the joint parameters and part segmentation, the authors introduced PatchImage Loss. Unlike standard image losses that look at the entire frame, PatchImage Loss calculates errors on small image patches. This makes the optimization process significantly more sensitive to fine-grained textures and edge alignments, preventing the "smearing" of details common in global loss functions.
Experimental Setup: The MMArt Dataset
To evaluate the framework, the researchers introduced the MMArt (Multi-Modal Articulated Objects Dataset). Built as an extension of the PartNet-Mobility dataset, MMArt includes 13 categories (such as toilets, dishwashers, and laptops) with 8 unique objects per category. The data was generated using Nvidia Isaac Sim with ray tracing to simulate realistic lighting and sensor noise, providing the multi-view RGB-D data necessary for testing interactive 3D reconstruction.
Results and Insights
SM³ demonstrates a significant leap over previous state-of-the-art (SOTA) methods like Ditto and Joint NeRF:
- Joint Accuracy: SM³ achieved an angular error of approximately 1.06°, a substantial improvement over Ditto’s 2.95°.
- Reconstruction Quality: In terms of movable part segmentation, SM³ showed a 96% improvement in Chamfer Distance, indicating a much higher fidelity in 3D shape recovery.
- Real-World Utility: The system was validated using an iPhone 14 to record everyday household objects. Despite the lack of controlled laboratory conditions, SM³ successfully reconstructed the geometry and joints of these items, proving its robustness to real-world sensor data.
Conclusion and Future Work
SM³ represents a major advancement in self-supervised robotic perception by treating physical interaction as a form of supervision. By asking, "How must this 3D part move to match the 2D observation?" the model gains a deep understanding of object mechanics.
However, some limitations remain. The current model is optimized for revolute joints (hinges) and does not yet account for prismatic joints (sliders, like drawers). Additionally, the method currently requires a "before" and "after" state; future iterations could involve the robot actively performing the interaction to gather its own training data. Despite these challenges, SM³ provides a scalable path toward robots that can learn to interact with the diverse and novel objects found in human homes.
Summary updated