Prompting Multi-Modal Image Segmentation with Semantic Grouping

Qibin HeAAAI 2024 · DOI

TL;DR

This paper introduces GoPT, a novel prompt tuning framework for multi-modal image segmentation that efficiently adapts large pre-trained foundation models. By employing "semantic grouping" through specialized prompt modules, GoPT balances modality-specific details with common cross-modal patterns, achieving state-of-the-art results with significantly fewer trainable parameters.

What Problem Does This Paper Address?

Multi-modal image segmentation, which combines information from various sensors (like RGB, depth, thermal, or radar) to create detailed scene understanding, is a crucial task in computer vision. The core challenge lies in effectively fusing information from different modalities. Existing methods often rely on extensive fine-tuning of large pre-trained models, which is computationally expensive, requires significant storage, and can struggle with limited downstream data due to poor transferability. Furthermore, these methods often face a dilemma: how to integrate common information across modalities while preserving unique, modality-specific patterns. This can lead to issues like "cross-modal gaps" (where the same object is described differently across modalities) and "information imbalances" (where one modality's contribution is over- or under-emphasized).

Methods

GoPT tackles these challenges by leveraging prompt learning, a technique that has shown great success in natural language processing. Instead of fine-tuning the entire foundation model, GoPT freezes the pre-trained model and introduces small, learnable "prompt tokens." The key innovation is the "semantic grouping" mechanism, implemented through two novel prompt modules:

  • Class-Aware Uni-Modal Prompter (CUP): This module focuses on balancing intra- and inter-modal semantic propagation. It uses modality-specific "class tokens" to group and aggregate features within a single modality, enhancing the representation of spatial information.
  • Alignment-Induced Cross-Modal Prompter (ACP): This module aims to aggregate class-aware representations and share prompt parameters across different modalities. It helps model common statistics and bridges the cross-modal gaps by aligning features.

These prompt modules are inserted hierarchically into the foundation model, allowing for efficient adaptation. The framework is designed to be general and applicable to various multi-modal segmentation tasks.

Datasets

The paper evaluates GoPT on several standard multi-modal image segmentation datasets:

  • NYUDv2: An indoor RGB-Depth (RGB-D) dataset.
  • SUN RGB-D: Another RGB-D dataset for indoor scene understanding.
  • MFNet: An urban street dataset for RGB-Thermal (RGB-T) segmentation.
  • PST900: A synchronized RGB-T dataset.
  • WHU-OS: A remote sensing dataset for RGB-Synthetic Aperture Radar (RGB-SAR) segmentation.

Key Insights and Ideas

  • Prompting for Multi-Modal Segmentation: The paper successfully demonstrates that prompt tuning, a parameter-efficient approach, can be effectively applied to multi-modal image segmentation, offering a compelling alternative to full fine-tuning.
  • Semantic Grouping as a Unifying Principle: The core idea of "semantic grouping" is central. By explicitly grouping features based on semantic similarity (both within and across modalities), GoPT can better handle the complexities of multi-modal data.
  • Balancing Specificity and Commonality: The dual-prompter design (CUP for modality-specific details and ACP for cross-modal commonalities) is a key insight into how to effectively fuse diverse sensor data.
  • Efficiency Gains: The significant reduction in trainable parameters (<1% of full fine-tuning) highlights the practical advantages of GoPT for deployment and training on resource-constrained environments.
  • Hard Assignment Superiority: The finding that "hard assignment" (assigning tokens to a single group) outperforms "soft assignment" in grouping suggests that a more exclusive grouping strategy leads to more discriminative prompts.

Key Results

GoPT achieves state-of-the-art (SOTA) performance on various downstream multi-modal image segmentation tasks. For instance:

  • On NYUDv2, GoPT (L) achieves 80.1% PAcc., 67.4% mAcc., and 52.3% mIoU, outperforming previous methods.
  • On SUN RGB-D, GoPT (L) achieves 85.5% PAcc., 64.6% mAcc., and 52.3% mIoU.
  • On MFNet, GoPT (L) achieves 77.0% mAcc. and 57.7% mIoU, surpassing the well-designed RGB-T segmenter RSFNet.
  • On WHU-OS, GoPT (B) achieves 72.9% OA, 69.3% AA, and 60.7% Kappa, outperforming the runner-up MIFNet.

Crucially, these results are achieved with a minimal number of trainable parameters (e.g., 0.97M for GoPT (B)), demonstrating a strong balance between performance and efficiency. Qualitative results show GoPT's ability to capture finer details and handle complex scenarios more accurately.

Limitations and Future Work

  • Limitations: The paper notes that while GoPT significantly reduces parameters, the "I" parameters (related to category settings) might still need fine-tuning for different datasets. The effectiveness of the prompt hierarchy is also explored, suggesting that more prompters generally lead to better performance, but the optimal number and placement might require further investigation.
  • Future Work: The authors suggest that the prompt tuning framework can be generalized to other vision tasks beyond segmentation. Further exploration into the optimal placement and number of grouping prompters could lead to even greater efficiency and performance gains. The potential for combining GoPT with other advanced prompt learning techniques also remains an open avenue.

Summary updated

BibTeX
@inproceedings{he2024prompting,
  author = {Qibin He},
  title = {{Prompting Multi-Modal Image Segmentation with Semantic Grouping}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i3.27981},
}
Figure