MarsSeg: Mars Surface Semantic Segmentation with Multi-level Extractor and Connector

Junbo Li, Keyan Chen, Gengju Tian, Lu Li, Zhenwei ShiarXiv 2024 · OpenReview

TL;DR

This paper introduces MarsSeg, a novel deep learning model for semantic segmentation of Martian terrain. MarsSeg addresses the challenges of low-contrast features and imbalanced datasets on Mars by employing a multi-level feature enhancement strategy within an encoder-decoder architecture, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Exploring Mars requires precise understanding of its surface for navigation, obstacle avoidance, and scientific analysis. However, segmenting Martian terrain is difficult due to several factors:

  • Complex Topography: Mars has varied terrain that can be hard to distinguish.
  • Similar Surface Features: Lack of vegetation and water leads to subtle differences between landforms, making it hard for algorithms to draw clear boundaries.
  • Imbalanced Datasets: Some Martian surface features (like "big rocks") are much rarer than others, making it challenging for models to learn them effectively.
  • Limited Annotated Data: Acquiring and annotating large datasets of Martian imagery is difficult, hindering the development of robust models.

Existing semantic segmentation methods, while successful on Earth, struggle with these unique Martian challenges.

Methods

MarsSeg is built upon a standard encoder-decoder architecture, but with key innovations to tackle the Martian terrain's complexities:

  • Minimized Down-sampling: The encoder uses fewer down-sampling layers than typical architectures. This helps preserve fine-grained local details and edge information, which are crucial for distinguishing subtle terrain features.
  • Feature Enhancement Connection Layer: This is the core novelty. Placed between the encoder and decoder, this layer intelligently fuses features from different levels. It comprises three modules:
  • Mini Atrous Spatial Pyramid Pooling (Mini-ASPP): A lightweight version of ASPP, designed to efficiently capture local multi-scale features. It uses smaller kernels and fewer dilation rates than standard ASPP to maintain computational efficiency on high-resolution feature maps. This is particularly useful for enhancing the representation of small objects and shadow features.
  • Polarized Self-Attention (PSA): This module combines channel and spatial attention mechanisms. It helps the model focus on relevant features across both dimensions, improving the understanding of global dependencies and enhancing the segmentation of large-scale terrain and shadow areas.
  • Strip Pyramid Pooling Module (SPPM): This module is designed for deep feature enhancement. It uses a combination of global adaptive pooling kernels and specialized "stripe" pooling kernels to capture contextual information from various scales and adapt to different terrain shapes.
  • Focal-Dice Loss Function: To address the class imbalance issue, the authors use a combined Focal loss and Dice loss. Focal loss down-weights easy examples and focuses on hard ones, while Dice loss is effective for imbalanced segmentation tasks. They also incorporate an "affinity loss" with adaptive weights to further balance the learning across categories.

Datasets

The paper uses two publicly available datasets for training and evaluation:

  • Mars-Seg: This dataset contains high-resolution RGB and grayscale images of Martian landscapes, divided into 9 categories (e.g., Martian soil, sands, gravel, bedrock, rocks, tracks, shadows, background, unknown). It was split into training and testing sets (8:2 ratio).
  • AI4Mars (MSL subset): This dataset, launched by NASA, includes a large collection of images from Mars rovers. The MSL subset used here has 16,064 training and 322 testing images, categorized into four types: soil, bedrock, sand, and big rock.

Key Insights and Ideas

  • Shadows are Informative: The paper highlights the importance of shadow features for segmentation, suggesting they provide crucial cues for understanding terrain geometry and distinguishing features, even in grayscale.
  • Lightweight Modules for High-Res: Applying complex modules like ASPP directly to high-resolution feature maps is computationally expensive. The Mini-ASPP and PSA are designed as efficient alternatives for shallow, high-resolution features.
  • Multi-Level Enhancement is Key: Combining modules that focus on local details (Mini-ASPP, PSA) with those that capture broader context (SPPM) is essential for robust Martian terrain segmentation.
  • Loss Function Matters for Imbalance: The custom loss function, incorporating Focal-Dice and affinity loss, is critical for achieving good performance on rare categories like "big rock."

Key Results

MarsSeg demonstrated state-of-the-art performance on both the Mars-Seg and AI4Mars datasets.

  • Mars-Seg (MSL-Seg): The proposed MarsSeg achieved a mean IoU (mIoU) of 65.69, significantly outperforming the baseline Deeplabv3+ (52.33) and other attention-based methods. It showed particular strength in categories like Tracks and Background.
  • Mars-Seg (MER-Seg): On the grayscale subset, MarsSeg achieved an mIoU of 52.48, also showing competitive results.
  • AI4Mars: MarsSeg achieved the highest mIoU (80.89) in the challenging "Big Rock" category, which has a very small sample proportion (less than 2%), highlighting its effectiveness in handling extreme class imbalance.
  • Ablation Studies: Experiments showed that each component (SPPM, PSA, Mini-ASPP, Balancing Loss) contributed positively to the overall performance, with the full MarsSeg model achieving the best results. The integration of Mini-ASPP was particularly effective for smaller objects like sands and rocks.

Limitations and Future Work

  • Computational Cost: While efforts were made to optimize modules, deep learning models for segmentation can still be computationally intensive.
  • Generalization: While the model performs well on the tested datasets, further validation on more diverse Martian terrain datasets would be beneficial.
  • Future Work: The authors suggest that their approach can be further adapted for other extraterrestrial exploration tasks and that the developed feature enhancement modules could be beneficial in other challenging segmentation scenarios.

Summary updated

Abstract

The segmentation and interpretation of the Martian surface play a pivotal role in Mars exploration, providing essential data for the trajectory planning and obstacle avoidance of rovers. However, the complex topography, similar surface features, and the lack of extensive annotated data pose significant challenges to the high-precision semantic segmentation of the Martian surface. To address these challenges, we propose a novel encoder-decoder based Mars segmentation network, termed MarsSeg. Specifically, we employ an encoder-decoder structure with a minimized number of down-sampling layers to preserve local details. To facilitate a high-level semantic understanding across the shadow multi-level feature maps, we introduce a feature enhancement connection layer situated between the encoder and decoder. This layer incorporates Mini Atrous Spatial Pyramid Pooling (Mini-ASPP), Polarized Self-Attention (PSA), and Strip Pyramid Pooling Module (SPPM). The Mini-ASPP and PSA are specifically designed for shadow feature enhancement, thereby enabling the expression of local details and small objects. Conversely, the SPPM is employed for deep feature enhancement, facilitating the extraction of high-level semantic category-related information. Experimental results derived from the Mars-Seg and AI4Mars datasets substantiate that the proposed MarsSeg outperforms other state-of-the-art methods in segmentation performance, validating the efficacy of each proposed component.
BibTeX
@misc{li2024marsseg,
  author = {Junbo Li and Keyan Chen and Gengju Tian and Lu Li and Zhenwei Shi},
  title = {{MarsSeg: Mars Surface Semantic Segmentation with Multi-level Extractor and Connector}},
  year = {2024},
  eprint = {2404.04155},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.04155},
}
Figure