Masked γ-SSL: Learning Uncertainty Estimation via Masked Image Modeling

David S. W. Williams, Matthew Gadd, Paul Newman, Daniele De Martini

TL;DR

This paper introduces Masked γ-SSL, a novel self-supervised learning approach that uses Masked Image Modeling (MIM) to train semantic segmentation networks capable of producing high-quality uncertainty estimates in a single forward pass. By leveraging general representations from foundation models and unlabelled domain-specific data, their method significantly improves robustness to distributional shifts, a critical challenge for safety-critical applications like autonomous driving.

What Problem Does This Paper Address?

Autonomous systems, particularly those used in safety-critical domains like self-driving cars, rely heavily on semantic segmentation to understand their surroundings. However, these networks often struggle when encountering data that differs from their training distribution (distributional shift). This can lead to incorrect segmentations, which are particularly dangerous when the system doesn't know it's making a mistake. The core problem this paper tackles is how to make semantic segmentation models not only accurate but also aware of their own limitations – providing reliable uncertainty estimates – especially when faced with unseen or slightly different environments. Existing methods for uncertainty estimation can be computationally expensive (requiring multiple forward passes) or sensitive to training hyperparameters.

Methods

The proposed method, Masked γ-SSL, builds upon a three-step training process:

  1. Pretraining: An encoder is trained to learn general visual representations from natural images using a foundation model like DINOv2. This step is performed once and provides a strong initialisation.
  2. Task Learning: The encoder is then used to train a semantic segmentation network (encoder + decoder) on a labelled "source" domain (e.g., Cityscapes). The encoder weights are frozen after this step.
  3. Uncertainty Training (The Core Contribution): This is where Masked γ-SSL shines. It uses unlabelled data from "target" domains (which may differ from the source domain) to refine the uncertainty estimation capabilities of the segmentation network. The key technique here is Masked Image Modeling (MIM). Instead of reconstructing pixel values, MIM is used to predict segmentation consistency. The network is trained to ensure its segmentation of masked image regions is consistent with the segmentation of the unmasked regions by a frozen, pre-trained network. Crucially, the loss is only applied to regions where the network is certain about its own prediction, effectively teaching it to be more confident when it's correct and to express uncertainty when it's likely to be wrong. This is achieved through a novel masking policy and an uncertainty masking procedure.

Datasets

  • Source Domain (Training): Cityscapes [38] (labelled)
  • Target Domains (Unlabelled Training & Labelled Testing):
    • SAX Segmentation benchmark: London (LDN), New Forest (NF), Scotland (SCOT) [2]
    • Berkeley DeepDrive (BDD) [39]
  • Test Dataset for Generalisation: WildDash [40] (labelled)

The SAX benchmark is highlighted as representing increasing distributional shifts relative to Cityscapes.

Key Insights and Ideas

  • MIM for Uncertainty, Not Just Representation: The paper cleverly repurposes MIM, typically used for learning general visual features, to specifically learn uncertainty estimation. By masking image patches and predicting segmentation consistency, the model learns to identify situations where its segmentation might be unreliable.
  • Leveraging Unlabelled Data is Key: The success of the method hinges on using unlabelled, domain-specific data. This allows the model to adapt to the nuances of new environments without requiring expensive manual annotation for every new scenario.
  • Certainty Guides Learning: The core idea of applying the loss only where the model is certain is a powerful way to refine uncertainty. It forces the model to improve its confidence in correct predictions and implicitly learn to express uncertainty in incorrect ones.
  • General Representations Matter for Uncertainty: The study shows that using general pre-trained encoders (like DINOv2) leads to better uncertainty estimation, even when the network is specialized for a specific task. This suggests that a broad understanding of visual concepts is beneficial for knowing what you don't know.
  • Single-Pass Efficiency: Unlike many ensemble-based uncertainty methods, Masked γ-SSL achieves high-quality uncertainty estimates in a single forward pass, making it practical for real-time robotic applications.

Key Results

  • Superior Uncertainty Estimation: The proposed Mask-d2 models consistently outperform various baselines (including epistemic uncertainty methods like Monte Carlo Dropout and deterministic methods like MaxS and GMM) on misclassification detection tasks, as measured by F0.5 scores and AUPR.
  • Robustness to Distributional Shift: The method shows significant performance gains on distributionally-shifted target domains (SAX datasets, WildDash) compared to baselines trained solely on the source domain (Cityscapes). For instance, on the WildDash benchmark, the Mask-BDD-d2 model performed best.
  • Generalisation: The models trained with Masked γ-SSL demonstrate good generalisation to unseen target domains.
  • Hyperparameter Robustness: The MIM approach is shown to be less sensitive to augmentation hyperparameters compared to Cross-and-Resize (C&R) augmentation methods.
  • Efficiency: The single-pass nature of the inference is a significant advantage for real-time systems.

Limitations and Future Work

  • Freezing Encoder: While freezing the encoder (fe) generally works well with sufficiently general representations (like DINOv2), the paper notes that for less general representations (like DINOv1), not freezing it can sometimes boost performance on specific target domains. This suggests a trade-off between generality and task-specific fine-tuning.
  • Masking Policy Simplicity: The paper uses a simple, independent masking policy. While effective, exploring more sophisticated masking strategies (e.g., learned policies) could potentially yield further improvements, though the current simplicity is a strength for generality.
  • Future Work: The paper implicitly suggests future work could involve exploring more complex masking policies and further investigating the interplay between encoder generality and task-specific fine-tuning for optimal uncertainty estimation across diverse domains. The success on the SAX benchmark also opens doors for further evaluation on more challenging and diverse real-world driving datasets.

Summary updated

Abstract

This work proposes a semantic segmentation network that produces high-quality uncertainty estimates in a single forward pass. We exploit general representations from foundation models and unlabelled datasets through a Masked Image Modeling (MIM) approach, which is robust to augmentation hyper-parameters and simpler than previous techniques. For neural networks used in safety-critical applications, bias in the training data can lead to errors; therefore it is crucial to understand a network’s limitations at run time and act accordingly. To this end, we test our proposed method on a number of test domains including the SAX Segmentation benchmark, which includes labelled test data from dense urban, rural and off-road driving domains. The proposed method consistently outperforms uncertainty estimation and Out-of-Distribution (OoD) techniques on this difficult benchmark.
BibTeX
@inproceedings{williams2024masked,
  author = {David S. W. Williams and Matthew Gadd and Paul Newman and Daniele De Martini},
  title = {{Masked γ-SSL: Learning Uncertainty Estimation via Masked Image Modeling}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=I91l27OIGD},
}
Figure