Masked Gamma-SSL: Learning Uncertainty Estimation via Masked Image Modeling

David S. W. Williams, Matthew Gadd, Paul Newman, Daniele De Martini

Abstract

This work proposes a semantic segmentation network that produces high-quality uncertainty estimates in a single forward pass. We exploit general representations from foundation models and unlabelled datasets through a Masked Image Modeling (MIM) approach, which is robust to augmentation hyper-parameters and simpler than previous techniques. For neural networks used in safety-critical applications, bias in the training data can lead to errors; therefore it is crucial to understand a network's limitations at run time and act accordingly. To this end, we test our proposed method on a number of test domains including the SAX Segmentation benchmark, which includes labelled test data from dense urban, rural and off-road driving domains. The proposed method consistently outperforms uncertainty estimation and Out-of-Distribution (OoD) techniques on this difficult benchmark.

Introduction

In autonomous driving, semantic segmentation—the ability to label every pixel in an image (e.g., "road," "pedestrian," "bus")—is critical. However, these networks often fail when they encounter "Out-of-Distribution" (OoD) scenarios, such as a dumpster they haven't seen before or a rural road that looks nothing like the urban training data.

For safety-critical robotics, it isn't enough for a network to be accurate; it must also be introspective. It needs to signal when it is uncertain so the system can take a safe action (like slowing down). Existing solutions like Monte Carlo Dropout or Ensembles are computationally expensive because they require multiple forward passes. This paper introduces Masked $\gamma$-SSL, a deterministic, single-pass method that provides reliable uncertainty estimates that generalize across different environments.

Methods

The researchers propose a three-step training pipeline:

  1. Pre-training: Utilizing a foundation model (DINOv2) to provide a robust, general representation of the world.
  2. Task Learning: Fine-tuning on a labelled "source" domain (e.g., Cityscapes) to learn the specific task of semantic segmentation.
  3. Uncertainty Training: Using unlabelled images from a "target" domain for a self-supervised consistency check.
The Masked Gamma-SSL training pipeline.

The core innovation is the use of Masked Image Modeling (MIM) as a stress test. The network is presented with two versions of the same image: one original and one where random patches are hidden (masked). A "frozen" version of the network segments the full image, while a "trainable" version segments the masked image. If the predictions differ significantly between the views, the model learns that these regions are likely sources of error. The loss function ($L_c$) encourages the model to increase its "certainty" only on pixels where these two views agree, effectively teaching the network to "know what it doesn't know."

Key Results

The study utilized a variety of datasets to test generalization, including Cityscapes (urban), the SAX Segmentation benchmark (London, New Forest, and Scotland), and BerkeleyDeepDrive (BDD). The proposed Mask-d2 model consistently outperformed baselines like Max-Softmax, GMMs, and even Ensembles on the SAX benchmark.

F0.5 scores across various benchmarks.

On the "Scotland" test set—the most significant shift from the training data—Mask-d2 achieved a Max $F_{0.5}$ score of 0.905, compared to 0.805 for a standard fine-tuned model (MaxS-d2). Qualitative results show the model correctly flagging "undefined" objects (like a dumpster or a cow in the road) as high-uncertainty areas in heatmaps, even when it attempts to incorrectly label them as "building" or "sidewalk."

Conclusion

Masked $\gamma$-SSL demonstrates that simple random patch masking is a powerful tool for learning uncertainty without the computational overhead of ensembles or the sensitivity of complex augmentations. By leveraging foundation models like DINOv2, the model retains general knowledge that prevents feature collapse during fine-tuning. Future work will investigate how this uncertainty can be directly integrated into the path-planning logic of autonomous vehicles to improve safety in unpredictable environments.

Summary updated

Abstract

This work proposes a semantic segmentation network that produces high-quality uncertainty estimates in a single forward pass. We exploit general representations from foundation models and unlabelled datasets through a Masked Image Modeling (MIM) approach, which is robust to augmentation hyper-parameters and simpler than previous techniques. For neural networks used in safety-critical applications, bias in the training data can lead to errors; therefore it is crucial to understand a network's limitations at run time and act accordingly. To this end, we test our proposed method on a number of test domains including the SAX Segmentation benchmark, which includes labelled test data from dense urban, rural and off-road driving domains. The proposed method consistently outperforms uncertainty estimation and Out-of-Distribution (OoD) techniques on this difficult benchmark.
BibTeX
@inproceedings{williams2024masked,
  author = {David S. W. Williams and Matthew Gadd and Paul Newman and Daniele De Martini},
  title = {{Masked Gamma-SSL: Learning Uncertainty Estimation via Masked Image Modeling}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=I91l27OIGD},
}
Figure