MARS: Multimodal Active Robotic Sensing for Articulated Characterization

Hongliang Zeng, Ping Zhang, Chengjiong Wu, Jiahua Wang, Tingyu Ye, Fang LiIJCAI 2024

Abstract

Accurate characterization of articulated objects is fundamental for robotic interaction in unstructured environments. We present MARS (Multimodal Active Robotic Sensing), a framework that integrates RGB and point cloud data through a Multi-Layer Dueling Module and a Transformer-based fusion block. To overcome occlusions and suboptimal viewpoints, MARS incorporates an Active Sensing module driven by Reinforcement Learning that autonomously repositions the camera based on a perception confidence score. Experimental results on the PartNet-Mobility dataset and real-world scenarios demonstrate that MARS achieves state-of-the-art performance, with active sensing reducing joint estimation errors by approximately 50%.

Introduction

For service robots to effectively interact with their environment—such as opening a cabinet or using a tool—they must precisely understand an object’s articulation. This involves identifying joint locations, movement types (e.g., revolute vs. prismatic), and current states. Current research often suffers from three major limitations: a reliance on single-modality data (ignoring visual cues like handles), the assumption of an "ideal viewpoint" that ignores occlusions, and the use of fragmented models for different joint types.

The MARS pipeline: from initial segmentation to active sensing and final manipulation.

MARS addresses these gaps by fusing multimodal data and introducing an active sensing strategy that allows the robot to autonomously seek better information when its initial perception is uncertain.

Methodology

The MARS framework is built upon two core components: Multimodal Feature Fusion Perception (MFFP) and Active Sensing (AS).

Multimodal Feature Fusion Perception (MFFP)

Rather than simple data concatenation, MARS utilizes a Multi-Layer Dueling Module (MLDM) to extract features from various layers of a ResNet backbone. This module employs a competitive mechanism to prioritize the most informative image scales, which are then fused with 3D point cloud data using a Transformer encoder. This approach allows the model to capture complex spatial relationships between 2D visual textures and 3D geometric structures.

Active Sensing (AS)

When the initial viewpoint yields a low "Perception Score" (a confidence metric), the Active Sensing module is triggered. Driven by a Deep Q-Network (DQN), the robot autonomously moves its camera to a more informative viewpoint. The reward function for this RL agent is optimized based on the improvement in perception confidence and the quality of the resulting point cloud data.

Detailed architecture of the Multimodal Feature Fusion Perception (MFFP) and Active Sensing (AS) modules.

Unified Joint Characterization

MARS standardizes the description of joint parameters, enabling a single neural network to predict position, orientation, and state for both revolute (hinged) and prismatic (sliding) joints simultaneously, simplifying deployment in general settings.

Experimental Results

The framework was evaluated using the SAPIEN simulator and the PartNet-Mobility dataset, covering 14 common object categories such as laptops, storage units, and scissors. Key findings include:

  • State-of-the-Art Accuracy: MARS outperformed existing benchmarks like Cart and EPNet across nearly all categories. It achieved a low error of 3.74° for revolute joint states and 0.07m for prismatic joint states.
  • The Impact of Active Sensing: The most significant result was that a single autonomous camera adjustment led to a ~50% reduction in perception error across all object categories. This confirms that where a robot looks is as critical as how it processes data.
  • Multimodal Synergy: Integrating multi-scale RGB features allowed the model to distinguish between geometrically similar parts by identifying visual markers like handles or decorative trim.
  • Robust Classification: The unified training approach resulted in 100% accuracy in identifying joint types (revolute vs. prismatic).

Conclusion

MARS demonstrates that combining multimodal fusion with active, confidence-driven sensing significantly enhances a robot's ability to characterize articulated objects. While the current system shows remarkable error reduction, future work will explore more advanced 3D backbones and increase the diversity of object categories to better reflect the complexity of real-world environments. Additionally, optimizing the active sensing search for faster, single-step viewpoint prediction remains a key area for development.

Summary updated

BibTeX
@inproceedings{zeng2024mars,
  author = {Hongliang Zeng and Ping Zhang and Chengjiong Wu and Jiahua Wang and Tingyu Ye and Fang Li},
  title = {{MARS: Multimodal Active Robotic Sensing for Articulated Characterization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure