Single Frame Semantic Segmentation Using Multi-Modal Spherical Images

Suresh Guttikonda, Jason RambachWACV 2024

TL;DR

This paper introduces a novel transformer-based architecture for semantic segmentation of 360° panoramic images, leveraging multiple sensor modalities (RGB, Depth, Normals, HHA) to overcome the challenges of distortion and limited data. Their approach achieves state-of-the-art performance on several indoor panoramic datasets by effectively fusing complementary information across modalities.

What Problem Does This Paper Address?

The increasing availability of 360° cameras has opened up new possibilities for comprehensive scene understanding, with applications ranging from augmented reality to autonomous driving. However, segmenting these panoramic images presents unique challenges:

  • Extreme Distortions: Equirectangular projections, commonly used for 360° images, introduce significant distortions, especially near the poles, making it difficult for standard computer vision models to process.
  • Data Scarcity: Labeled datasets for panoramic semantic segmentation are scarce, hindering the training of robust models.
  • Complementary Modalities: While RGB data provides rich texture information, it can be ambiguous in certain scenarios. Other modalities like depth and normals offer geometric cues that can disambiguate these situations, but effectively fusing them for panoramic images is an open research problem.

Existing research has largely focused on pinhole camera models or RGB-only panoramic segmentation. This paper aims to bridge this gap by developing a multi-modal fusion strategy specifically designed for the challenges of equirectangular panoramic images.

Methods

The authors propose a transformer-based cross-modal fusion architecture that addresses both panoramic distortions and multi-modal integration. Key components include:

  • Panoramic Hierarchical Encoder: This encoder processes the input image in stages, progressively extracting features at different resolutions. It incorporates:
  • Deformable Patch Embeddings (DPE): Unlike standard patch embeddings that use fixed sampling points, DPE learns data-dependent offsets to better capture object deformations and distortions inherent in panoramic images.
  • Cross-Modal Feature Rectification (CM-FRM): This module addresses noisy or uncertain information from different sensor modalities by using channel-wise and spatial-wise rectification to calibrate features. It extends previous work to handle tri-modal inputs (RGB, Depth, Normals).
  • Cross-Modal Feature Fusion (FFM): A two-stage module that uses ternary multi-head cross-attention to facilitate global information flow and fuse features from different modalities.
  • Panoramic Token Mixer Decoder: This decoder leverages the DMLPv2 module, which is designed for efficient and adaptive spatial and channel-wise token mixing, to handle object deformations more effectively than standard MLP decoders.

The architecture is designed to be flexible, supporting bi-modal and tri-modal input scenarios.

Datasets

The authors evaluate their approach on three indoor panoramic datasets:

  • Stanford2D3DS [2]: Contains 1713 multi-modal panoramic images with 13 object categories.
  • Structured3D [44]: Offers 196,515 synthetic multi-modal panoramic images with 40 object categories.
  • Matterport3D [5]: A dataset of 10,800 panoramic views, which are converted to an equirectangular format.

They also mention that the HHA modality for Stanford2D3DS is computed using publicly available code.

Key Insights and Ideas

  • Deformable Embeddings are Crucial: The paper highlights that standard patch embeddings are insufficient for panoramic images due to severe distortions. Employing deformable embeddings is key to learning robust, distortion-aware features.
  • Multi-Modal Fusion Significantly Boosts Performance: Combining different sensor modalities (RGB, Depth, Normals, HHA) consistently improves segmentation accuracy, especially in challenging scenarios where RGB alone is ambiguous.
  • Cross-Modal Interactions are Key for Calibration: The CM-FRM module is vital for cleaning up and aligning features from different modalities, preventing noisy data from degrading performance.
  • Normals Add Value, Even with Depth: While depth provides strong geometric cues, adding normals further refines segmentation, particularly for fine structures.
  • Potential for Modality Redundancy: In some cases, particularly with Matterport3D, the authors observe that adding more modalities (beyond RGB-D) doesn't always yield significant improvements, suggesting potential redundancy or limitations in how modalities are combined.

Key Results

The proposed method achieves state-of-the-art performance on the evaluated datasets:

  • Stanford2D3DS [2]:
    • 60.60% mIoU with RGB-HHA.
  • 71.97% mIoU with RGB-D-N on Structured3D.
  • 35.92% mIoU with RGB-D on Matterport3D.
  • Structured3D [44]: Achieved 71.94% mIoU for RGB-only segmentation, outperforming benchmarks. With RGB-Depth-Normals fusion, they reached 75.86% mIoU.
  • Matterport3D [5]: Achieved 35.15% mIoU for RGB-only and 39.19% mIoU for RGB-D.

Qualitative results demonstrate that their approach produces clearer geometric shapes and more accurate object semantics compared to baseline methods, even handling thin structures effectively.

Limitations and Future Work

  • Computational Complexity: The authors acknowledge that using multiple input streams can lead to a significant increase in computational complexity.
  • Modality Redundancy: For some datasets (e.g., Matterport3D), adding more modalities beyond RGB-D did not lead to expected improvements, suggesting that modal addition might not always be necessary or that the fusion strategy could be further optimized.
  • Future Work: The authors plan to explore combining multi-modal panoramas with 3D LiDAR data for even more comprehensive scene understanding, aiming to do so with minimal processing effort.

Summary updated

BibTeX
@inproceedings{guttikonda2024single,
  author = {Suresh Guttikonda and Jason Rambach},
  title = {{Single Frame Semantic Segmentation Using Multi-Modal Spherical Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure