IFE-CMT: Instance-Aware Fine-Grained Feature Enhancement Cross Modal Transformer for 3D Object Detection

Xiaona Song, Haonan Zhang, Haichao Liu, Xinxin Wang, Lijun WangSensors 2025 · DOI

TL;DR

This paper introduces IFE-CMT, a novel multi-modal 3D object detection model that significantly improves the detection of small and easily confused objects. By introducing an "Instance feature Enhancement Module" (IE-Module) and an improved point cloud backbone, IFE-CMT achieves better fine-grained feature representation, leading to higher overall detection accuracy.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurate 3D object detection to perceive their surroundings. While multi-modal approaches (combining camera images and LiDAR point clouds) have shown promise, existing methods often focus on coarse fusion strategies. This neglect of finer-grained details leads to a significant drop in detection accuracy for small objects (like bicycles and motorcycles) and objects with similar appearances, which are crucial for safe navigation. The paper aims to bridge this gap by enhancing the model's ability to capture and leverage these subtle, instance-specific features.

Methods

The core of IFE-CMT lies in two main innovations:

  • Instance feature Enhancement Module (IE-Module): This module is designed to extract and utilize instance-level features from multi-modal data. It uses an "instance heatmap" to pinpoint object locations and then employs a self-learning decoder and attention mechanisms to fuse these instance features with broader scene features. This process enhances the point cloud features with rich semantic information from images, leading to a more refined representation without explicit view transformations. A Gaussian Focal Loss is used to train this module.
  • Improved Point Cloud Branch: The authors redesign the point cloud processing backbone. They combine the strengths of Feature Pyramid Networks (FPN) for multi-scale feature extraction with Atrous Spatial Pyramid Pooling (ASPP) to expand the receptive field and capture richer contextual information. Additionally, Spatial and Channel Reconstruction Convolution (SCConv) modules are integrated to reduce feature redundancy and improve computational efficiency.

Datasets

The model was trained and evaluated on the nuScenes dataset. This is a large-scale, open-source dataset widely used for multi-modal 3D object detection, featuring data from cameras, LiDAR, radar, IMU, and GPS. The authors specifically used camera and LiDAR data for their experiments.

Key Insights and Ideas

  • Fine-grained features are critical for small objects: The paper strongly emphasizes that focusing solely on overall scene fusion is insufficient. The ability to discern subtle, instance-specific details is paramount for accurately detecting small or occluded objects.
  • Instance heatmaps as a powerful tool: Using instance heatmaps to guide feature extraction and enhancement proves effective in localizing and enriching object representations.
  • Synergy of FPN and ASPP for point clouds: Combining FPN's multi-scale capabilities with ASPP's expanded receptive field offers a more robust approach to point cloud feature extraction than either alone, especially in the context of 3D data where scale variations are less perspective-dependent.
  • SCConv for efficiency: The introduction of SCConv effectively tackles feature redundancy, leading to a more efficient network without sacrificing performance.

Key Results

The IFE-CMT model demonstrated significant improvements over the baseline CMT model on the nuScenes dataset:

  • Overall Performance: On the validation set, IFE-CMT achieved a 2.1% increase in mean Average Precision (mAP) and a 0.8% increase in nuScenes Detection Score (NDS). On the test set, these improvements were 1.9% and 0.7%, respectively.
  • Small Object Detection: Notably, the model showed substantial gains for small object categories: mAP improved by 6.6% for bicycles and 3.7% for motorcycles.
  • Qualitative Results: Visualizations showed IFE-CMT successfully detecting occluded motorcycles and accurately separating crowded pedestrians, tasks where the baseline struggled.

Limitations and Future Work

The authors acknowledge that while their model achieves state-of-the-art performance, direct comparison with all SOTA models is limited by current GPU configurations and scale. Future work will focus on:

  • Exploring more powerful backbone networks: Investigating architectures like ResNet-101 and Swin-Transformer, and higher input resolutions.
  • Lightweight architectural design: Further optimizing the model for real-time inference while maintaining detection accuracy.
  • Temporal fusion: Examining the scalability and effectiveness of the IE-Module when incorporated into temporal fusion models for video-based detection.

Summary updated

Abstract

In recent years, multi-modal 3D object detection algorithms have experienced significant development. However, current algorithms primarily focus on designing overall fusion strategies for multi-modal features, neglecting finer-grained representations, which leads to a decline in the detection accuracy of small objects. To address this issue, this paper proposes the Instance-aware Fine-grained feature Enhancement Cross Modal Transformer (IFE-CMT) model. We designed an Instance feature Enhancement Module (IE-Module), which can accurately extract object features from multi-modal data and use them to enhance overall features while avoiding view transformations and maintaining low computational overhead. Additionally, we design a new point cloud branch network that effectively expands the network's receptive field, enhancing the model's semantic expression capabilities while preserving texture details of the objects. Experimental results on the nuScenes dataset demonstrate that compared to the CMT model, our proposed IFE-CMT model improves mAP and NDS by 2.1% and 0.8% on the validation set, respectively. On the test set, it improves mAP and NDS by 1.9% and a 0.7%. Notably, for small object categories such as bicycles and motorcycles, the mAP improved by 6.6% and 3.7%, respectively, significantly enhancing the detection accuracy of small objects.
BibTeX
@inproceedings{song2025ife,
  author = {Xiaona Song and Haonan Zhang and Haichao Liu and Xinxin Wang and Lijun Wang},
  title = {{IFE-CMT: Instance-Aware Fine-Grained Feature Enhancement Cross Modal Transformer for 3D Object Detection}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25185685},
}
Figure