SM: Self-supervised Multi-task Modeling with Multi-view 2D Images for Articulated Objects

Haowen Wang, Zhen Zhao, Zhao Jin, Zhengping Che, Liang Qiao, Yakun Huang, Zhipeng Fan, Xiuquan Qiao, Jian Tang

TL;DR

This paper introduces SM³, a self-supervised method that reconstructs textured 3D models of articulated objects and estimates their joint parameters using only multi-view 2D images captured before and after interaction. This approach eliminates the need for extensive 3D annotations and significantly advances the field of robotic manipulation and virtual environment creation.

What Problem Does This Paper Address?

Reconstructing the 3D geometry and understanding the articulation (how parts move) of real-world objects is crucial for robotics and virtual reality. Existing methods often rely on heavily annotated 3D datasets, which are expensive and time-consuming to create, and struggle with the diversity of objects found in the real world. Supervised approaches are limited to specific object categories and require precise 3D models and joint information. While some recent methods have made progress by using multi-view images or focusing on interaction, they still face challenges in accurately modeling novel objects, estimating motion states, and require costly datasets with detailed annotations. This paper aims to bridge this gap by proposing a self-supervised method that can achieve high-quality reconstruction and articulation estimation from readily available multi-view 2D images.

Methods

The core of the proposed method, SM³, is a self-supervised, multi-task learning framework. It leverages multi-view RGB images taken before and after an object's interaction (e.g., a part being moved).

Here's a breakdown of the key components:

  • 3D Reconstruction (based on Nvdiffrec): The method starts by using a differentiable tetrahedral grid approach, similar to Nvdiffrec, to reconstruct the object's geometry and texture. This process is optimized by minimizing the difference between rendered images from the reconstructed model and the input multi-view images. This is done for both the pre-interaction and post-interaction states, resulting in two 3D models: $M_{pre}$ and $M_{post}$.
  • Movable Part Segmentation Prior: To identify which parts of the object can move, SM³ analyzes the geometric differences between $M_{pre}$ and $M_{post}$. Vertices in the pre-interaction model are classified as "movable" if they are significantly distant from the post-interaction model, and "static" otherwise. This provides a prior for segmenting the movable part.
  • Candidate Joints Prediction: The method assumes that joint directions often align with the object's primary planes. It uses Principal Component Analysis (PCA) on the static part's point cloud to establish a local coordinate system. By analyzing bounding boxes of movable and static parts, it identifies candidate joint positions and directions.
  • Integrated Optimization: This is the crucial self-supervised step. The movable components are rigidly rotated using the predicted joint parameters (rotation axis and pivot point). The system then renders the rotated object and compares it to the post-interaction images, calculating an "interaction loss." This loss, along with losses for joint position, direction, and segmentation, is used to jointly optimize the movable part segmentation and joint parameters. A "PatchImage Loss" is introduced to refine the image loss by processing image patches, making it more robust to noise and misalignments.

Datasets

  • MMArt Dataset: The authors introduce a new dataset called MMArt. It's an extension of the PartNet-Mobility dataset, focusing on multi-view, multi-modal, and multi-state data of articulated objects. It includes 13 categories and 8 objects per category, generated using Nvidia's Isaac Sim with realistic lighting and sensor simulation (Intel RealSense D435). This dataset is designed to address the lack of comprehensive multi-view 2D image data for articulated object reconstruction.

Key Insights and Ideas

  • Leveraging Interaction for Self-Supervision: The core innovation is using the difference between pre- and post-interaction states as a powerful self-supervisory signal. This allows the model to learn about motion and articulation without explicit labels.
  • Joint Optimization of Geometry, Segmentation, and Motion: Instead of treating these as separate problems, SM³ jointly optimizes them. This synergy allows geometric priors to inform segmentation and motion estimation, and vice-versa, leading to more accurate results.
  • Geometric Priors from Pre/Post Analysis: Deriving segmentation priors and candidate joint information directly from the geometric discrepancies between the two states is a clever way to inject domain knowledge without manual annotation.
  • Robustness through Patch-based Loss: The PatchImage Loss is a practical improvement for handling real-world image imperfections and improving the granularity of the optimization.

Key Results

  • State-of-the-Art Performance: SM³ significantly outperforms existing benchmarks across various categories and objects in terms of both joint parameter estimation (lower angle and position errors) and 3D reconstruction (lower Chamfer Distance for movable parts).
  • Improved Movable Part Segmentation: The method achieves a distinct advantage in movable part segmentation, with up to a 96% improvement in Chamfer Distance compared to the second-best method (Ditto).
  • Real-World Validation: The method demonstrates strong performance on real-world captured data, showcasing its practical applicability.
  • Ablation Studies: Experiments confirm the importance of each component: segmentation priors, candidate joints, and the PatchImage Loss all contribute significantly to the overall performance.

Limitations and Future Work

  • Limitations:
    • The method assumes a single rotational joint per object for simplicity in the current formulation.
  • While robust, the accuracy of keypoint detection can still influence performance in some scenarios, as noted with the Joint NeRF baseline.
  • The paper mentions that Ditto [18] still misplaces joint direction for some objects due to intra-class morphological variations, implying that SM³ might also face similar challenges with highly complex or ambiguous object structures.
  • Future Work:
    • The authors suggest that the MMArt dataset can be easily adapted for single-object or single-modal datasets, implying future work could explore these specialized scenarios.
  • The success of the self-supervised approach opens doors for extending it to more complex articulation types (e.g., prismatic joints) and more intricate object interactions.

Summary updated

Abstract

Reconstructing real-world objects and estimating their movable joint structures are pivotal technologies within the field of robotics. Previous research has predominantly focused on supervised approaches, relying on annotated datasets to model articulated objects within limited categories. However, these approaches fall short of effectively addressing the diversity present in the real world. To tackle this issue, we propose a self-supervised interaction perception method, referred to as SM 3 , which leverages multi-view RGB images captured before and after interaction to model articulated objects, identify the movable parts, and infer the parameters of their rotating joints. By constructing 3D geometries and textures from the captured 2D images, SM 3 achieves integrated optimization of movable part and joint parameters during the reconstruction process, obviating the need for annotations. Furthermore, we introduce the MMArt dataset, an extension of PartNet-Mobility, encompassing multi-view and multi-modal data of articulated objects spanning diverse categories. Evaluations demonstrate that SM 3 surpasses existing benchmarks across various categories and objects, and its adaptability in real-world scenarios has been thoroughly validated.
BibTeX
@inproceedings{wang2024sm,
  author = {Haowen Wang and Zhen Zhao and Zhao Jin and Zhengping Che and Liang Qiao and Yakun Huang and Zhipeng Fan and Xiuquan Qiao and Jian Tang},
  title = {{SM: Self-supervised Multi-task Modeling with Multi-view 2D Images for Articulated Objects}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=jU26c7dBUW},
}
Figure