FaultSeg Swin-UNETR: Transformer-Based Self-Supervised Pretraining Model for Fault Recognition

Zeren Zhang, Ran Chen, Jinwen Ma

TL;DR

This paper introduces FaultSeg Swin-UNETR, a novel self-supervised pre-training approach for seismic fault recognition. By leveraging the SimMIM pre-training task on a large unlabeled seismic dataset and refining the Swin-UNETR architecture with multi-scale decoding, the model achieves state-of-the-art performance on the challenging Thebe dataset, overcoming limitations of existing methods.

What Problem Does This Paper Address?

Seismic fault interpretation is crucial for understanding Earth's subsurface, aiding in geological research, resource exploration, and hazard assessment. However, traditional manual interpretation is subjective and time-consuming. Automated methods, while promising, often struggle with data quality, quantity, and generalization issues, especially when trained on limited synthetic data and applied to real-world seismic surveys. These real-world datasets are characterized by their complexity, noise, and the inherent difficulty and cost of obtaining accurate fault annotations, leading to significant performance degradation for supervised deep learning models.

Methods

The core of this work lies in a two-pronged approach: self-supervised pre-training and a refined network architecture.

  1. Self-Supervised Pre-training (SimMIM): The authors adopt the Simplified Masked Image Modeling (SimMIM) technique, originally successful in natural image processing. They apply this to a large corpus of unlabeled 3D seismic data using a 3D Swin Transformer as the backbone. SimMIM involves masking a significant portion of the input data and training the network to reconstruct the missing parts. This allows the model to learn rich, domain-specific features without relying on manual annotations.
  2. Refined Swin-UNETR Architecture: For the fine-tuning stage, the authors adapt the Swin-UNETR architecture. Recognizing the narrow and sparse nature of seismic faults, they introduce:
  3. Multi-Scale Decoding: The network employs three parallel decoders, each trained to reconstruct faults at different dilation levels (1x, 3x, 5x). This helps capture faults of varying widths and extents.
  4. Fusion Module: A dedicated fusion module is used to combine the outputs from the multi-scale decoders and restore the original fault thickness, addressing the issue of dilated fault predictions.
  5. Whole Network Pre-training: Crucially, they found that pre-training the entire Swin-UNETR segmentation network (not just the backbone) with SimMIM yields superior results. This contrasts with typical self-supervised approaches that focus solely on backbone pre-training and is attributed to the network's specific parameter distribution.

Datasets

  • Unlabeled Seismic Data: A "large amount" of unlabeled real seismic data from 15 different working areas was used for the SimMIM pre-training phase. The specific size and origin of this data are not detailed beyond its use for learning domain-specific features.
  • Thebe Dataset: This is the primary dataset used for fine-tuning and evaluation. It's described as the "largest currently publicly available real seismic fault dataset," originating from the Thebe Gas Field in Australia. The dataset is provided in Python Numpy format with dimensions of 1803 × 1537 × 3174. For training, it was split along the crossline direction: 900 slices for training, 200 for validation, and 703 for testing.
  • Synthetic Dataset: A synthetic dataset of 220 seismic volumes (128x128x128) was also used for ablation studies to demonstrate the effectiveness of pre-training under limited data conditions.

Key Insights and Ideas

  • SimMIM is effective for seismic data: The SimMIM pre-training task proves to be a powerful tool for learning seismic domain features from unlabeled data, significantly boosting downstream performance.
  • Whole network pre-training is key: Pre-training the entire Swin-UNETR segmentation network, rather than just its backbone, leads to substantial performance gains. This suggests that for this architecture and task, learning from the decoder's perspective during pre-training is highly beneficial.
  • Multi-scale decoding addresses fault sparsity: The novel multi-decoder approach effectively handles the challenge of narrow and sparse fault structures by learning representations at different scales.
  • Swin Transformer's suitability: The Swin Transformer, with its hierarchical and window-based attention, is well-suited for seismic data, offering better parameter efficiency compared to other Transformer variants for this task.

Key Results

The proposed FaultSeg Swin-UNETR model achieved state-of-the-art performance on the Thebe dataset, as measured by the OIS (One-Image-best threshold) and ODS (Overall-best threshold) metrics.

  • The final model achieved OIS of 0.875 and ODS of 0.870.
  • This performance surpasses previous methods, including DeeplabV3 with ImageNet pre-training and other Swin-UNETR variants.
  • Ablation studies showed that self-supervised pre-training on the backbone improved performance over a non-pretrained Swin-UNETR, but pre-training the entire network yielded the best results.
  • Even when fine-tuned on a small synthetic dataset, the pre-trained Swin-UNETR model outperformed the FaultSeg3D model.

Limitations and Future Work

  • Data Scarcity: The authors acknowledge that the field of seismic fault identification still suffers from significant data scarcity, even with the Thebe dataset.
  • Future Direction: They suggest the urgent need for a "data-driven unified model similar to SAM" that can leverage semi-automatic annotation to expand fault datasets, paving the way for a truly universal fault recognition AI.

Summary updated

Abstract

This paper introduces an approach to enhance seismic fault recognition through self-supervised pretraining. Seismic fault interpretation holds great significance in the fields of geophysics and geology. However, conventional methods for seismic fault recognition encounter various issues, including dependence on data quality and quantity, as well as susceptibility to interpreter subjectivity. Currently, automated fault recognition methods proposed based on small synthetic datasets experience performance degradation when applied to actual seismic data. To address these challenges, we have introduced the concept of self-supervised learning, utilizing a substantial amount of relatively easily obtainable unlabeled seismic data for pretraining. Specifically, we have employed the Swin Transformer model as the core network and employed the SimMIM pretraining task to capture unique features related to discontinuities in seismic data. During the fine-tuning phase, inspired by edge detection techniques, we have also refined the structure of the Swin-UNETR model, enabling multiscale decoding and fusion for more effective fault detection. Experimental results demonstrate that our proposed method attains state-of-the-art performance on the Thebe dataset, as measured by the OIS and ODS metrics.
BibTeX
@misc{zhang2023faultseg,
  author = {Zeren Zhang and Ran Chen and Jinwen Ma},
  title = {{FaultSeg Swin-UNETR: Transformer-Based Self-Supervised Pretraining Model for Fault Recognition}},
  year = {2023},
  eprint = {2310.17974},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.17974},
}
Figure