Scribble-Based Domain Adaptation via Co-segmentation

Reuben Dorent, Samuel Joutard, Jonathan Shapey, Sotirios Bisdas, Neil Kitchen, Robert Bradford, Shakeel Saeed, Marc Modat, Sébastien Ourselin, Tom Vercauteren

TL;DR

Deep learning models for medical image segmentation struggle to generalize across different imaging modalities (e.g., from T1 to T2 MRI). This paper introduces a weakly-supervised domain adaptation method that uses simple "scribble" annotations on the target domain, formulated as a co-segmentation problem with a novel regularized loss. The approach achieves performance comparable to—and even slightly better than—fully supervised methods, while drastically reducing the time required for human annotation.

What Problem Does This Paper Address?

While Deep Neural Networks (DNNs) achieve state-of-the-art results in medical image segmentation, they are notoriously brittle when faced with a "domain shift." A model trained to segment tumors on a contrast-enhanced T1 MRI will likely fail when applied to a T2 MRI.

To fix this, researchers typically use Domain Adaptation (DA). The two standard approaches both have major flaws:

  1. Supervised DA: Requires clinicians to painstakingly draw full boundary annotations on the new target data, which is expensive and time-consuming.
  2. Unsupervised DA (UDA): Uses adversarial training or self-supervision to align domains without new labels. However, these methods are highly unstable, rely on complex heuristics, and often fail when the visual differences between domains are severe.

This paper seeks a practical middle ground: Weakly-supervised Domain Adaptation. Can we adapt a model to a new modality using only quick, rough "scribbles" from a clinician, rather than exact boundaries?

Methods

The authors frame domain adaptation as a dataset-level co-segmentation problem using a Conditional Random Field (CRF).

Here is how the system is built:

  • The Graphical Model: They construct a graph where every voxel (3D pixel) in every image is a node. Edges connect voxels within the same image (to ensure local spatial and visual consistency) and between different images (to transfer knowledge from the fully-labeled source domain to the scribble-labeled target domain).
  • Bridging the Domain Gap: Normally, CRFs link pixels with similar intensities. But a tumor looks completely different in T1 vs. T2 scans. To solve this, the authors compute the cross-image similarity using the deep features extracted from the network's penultimate layer, rather than raw image intensities.
  • A Single-Step Regularized Loss: Traditionally, training a CRF alongside a neural network requires a slow, alternating two-step process (generate label proposals $\rightarrow$ train network $\rightarrow$ repeat). The authors mathematically relax this problem, approximating the CRF optimization as a direct, single-step regularized loss function. This makes the model stable and easy to train end-to-end using standard gradient descent.

Datasets

The framework is evaluated on the segmentation of Vestibular Schwannoma (VS), a type of benign brain tumor.

  • Source Domain: 150 fully annotated contrast-enhanced T1-weighted (T1-c) scans.
  • Target Domain (Training): 30 T2 scans annotated only with scribbles. On average, only 1% of the total scan and 7% of the tumor volume was annotated.
  • Target Domain (Testing): 50 T2 scans that were manually, fully segmented to serve as the ground truth for evaluation.

Key Insights and Ideas

  • Scribbles succeed where unsupervised methods fail: Because the anatomical appearance of the inner brain varies so drastically between T1 and T2 scans, standard unsupervised adversarial methods completely collapsed. A tiny amount of human guidance (scribbles) was enough to bridge this massive domain gap.
  • Feature-based affinity is crucial for cross-modality DA: By using the network's learned feature representations to establish relationships between source and target images, the model successfully groups similar tissues together, even when their raw pixel colors/intensities are entirely different.
  • Mathematical relaxation speeds up training: By converting the complex CRF proposal generation into a direct loss penalty, the authors created a method that is computationally efficient and avoids the instability of adversarial training.

Key Results

The quantitative results (measured by Dice score, where higher is better, and Average Symmetric Surface Distance (ASSD), where lower is better) are striking:

  • Baseline (Source only): 28.2% Dice (Shows the severity of the domain shift).
  • Unsupervised DA baselines: 9.3% to 27.3% Dice (Complete failure).
  • Target Scribbles Only: 77.6% Dice.
  • Proposed Method (Scribbles + Source + Domain Adaptation Loss): 83.4% Dice and 0.8 mm ASSD.
  • Fully Supervised Target (Upper Bound): 81.6% Dice.

Crucially, the proposed weakly-supervised method slightly outperformed a model trained on fully annotated target data. This proves that combining rich, fully-labeled source data with sparse, cheaply obtained target data is an incredibly powerful paradigm.

Limitations and Future Work

While the authors do not dedicate a specific section to limitations, a clear technical hurdle is computational complexity. Connecting every voxel in a dataset to every other voxel creates a massive quadratic computation bottleneck. The authors mitigated this by using high-dimensional filtering and randomly sampling feature channels during training to keep memory and compute costs linear.

Future work could explore applying this framework to multi-class segmentation problems, or testing it across even wider domain gaps (e.g., adapting from CT to MRI).

Summary updated

Abstract

Although deep convolutional networks have reached state-of-the-art performance in many medical image segmentation tasks, they have typically demonstrated poor generalisation capability. To be able to generalise from one domain (e.g. one imaging modality) to another, domain adaptation has to be performed. While supervised methods may lead to good performance, they require to fully annotate additional data which may not be an option in practice. In contrast, unsupervised methods don’t need additional annotations but are usually unstable and hard to train. In this work, we propose a novel weakly-supervised method. Instead of requiring detailed but time-consuming annotations, scribbles on the target domain are used to perform domain adaptation. This paper introduces a new formulation of domain adaptation based on structured learning and co-segmentation. Our method is easy to train, thanks to the introduction of a regularised loss. The framework is validated on Vestibular Schwannoma segmentation (T1 to T2 scans). Our proposed method outperforms unsupervised approaches and achieves comparable performance to a fully-supervised approach.
BibTeX
@inproceedings{dorent2020scribble,
  author = {Reuben Dorent and Samuel Joutard and Jonathan Shapey and Sotirios Bisdas and Neil Kitchen and Robert Bradford and Shakeel Saeed and Marc Modat and Sébastien Ourselin and Tom Vercauteren},
  title = {{Scribble-Based Domain Adaptation via Co-segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  url = {https://openreview.net/forum?id=oRoZmJqEvs3},
}

Loading related content...

Figure