Cut out the annotator, keep the cutout: better segmentation with weak supervision

Sarah M. Hooper, Michael Wornow, Ying Hang Seah, Peter Kellman, Hui Xue, Frederic Sala, Curtis P. Langlotz, Christopher Ré

Introduction

Deep learning has revolutionized image segmentation, but it suffers from a severe "data hunger" problem. In domains like medical imaging, this is particularly acute: segmenting 3D or 4D volumes requires expensive expert time, and a model trained for one scanner or anatomy rarely generalizes to another.

Current "low-data" solutions have significant gaps. Few-Shot Learning (FSL) methods, such as data augmentation or transfer learning, are often brittle—their performance fluctuates wildly depending on which specific images are chosen for labeling. On the other hand, Weak Supervision (WS) typically relies on "labeling functions" (LFs) like hand-written rules or heuristics. While effective for text, it is incredibly difficult to write a program that accurately describes the visual texture of a complex organ or tumor across thousands of noisy pixels.

To bridge this gap, researchers have introduced a framework that fuses FSL and WS to achieve high-performance segmentation using as few as five labeled images. By using one-shot models as "weak sources" and a novel "contention-aware" graphical model to aggregate their outputs, the method achieves results within 1.4 Dice points of fully supervised networks.

The Proposed Pipeline

The researchers propose a three-step pipeline that intelligently manages noise while minimizing manual labeling:

1. Generating Weak Labels (The FSL Phase) Instead of writing manual rules, the team takes just five manually labeled images and trains five separate "one-shot" Convolutional Neural Networks (CNNs). Each CNN acts as a Labeling Function (LF) that provides a noisy, preliminary segmentation for the rest of the unlabeled dataset.

Conceptual illustration of Step 1: Using noisy labeling functions to generate initial masks for unlabeled data.

2. The Contention-Based Label Model (The WS Phase) This step represents the core innovation of the paper. Standard WS models assign a single "accuracy" score to a labeling source. However, in segmentation, a model might be 99% accurate on the easy background but 0% accurate at the complex organ boundary. To solve this, the authors developed a Probabilistic Graphical Model (PGM) that focuses on "areas of contention"—pixels where the five one-shot models disagree. The PGM learns to weight the sources differently in these high-conflict zones without needing ground-truth labels.

3. Training the End Model The aggregated "consensus" labels generated by the PGM are then used to train a final, robust segmentation network (a U-Net). This final model is subsequently fine-tuned on the original five labeled images to maximize precision.

Key Insights

Accuracy is a Coarse Metric The authors highlight that global accuracy is a fundamentally flawed way to evaluate a labeling function for images. Because the background often makes up the vast majority of an image, a model can achieve high accuracy simply by predicting "background" everywhere, failing completely where it matters most: the edges of the subject.

Visualizing the flaw of global accuracy: A tiny mask (b) or a massive circle (c) can achieve high accuracy simply by correctly guessing the large background area, despite failing entirely at the actual boundaries.

Contention as a Signal By tying model parameters to "agreement conditions" (e.g., determining how to weight Model A when it disagrees with Models B and C), the system can more finely track performance at difficult segmentation borders.

FSL Models Make Better LFs One-shot CNNs are vastly superior at capturing rich image information compared to the simple heuristics (like intensity thresholds) typically used in traditional weak supervision.

Results and Performance

The method was rigorously evaluated across seven different medical segmentation tasks using three major public datasets: ACDC (Cardiac MRI), Abdominal CT, and BRATS (Head MRI).

  • Near-Supervised Performance: With only five labeled images, the proposed method achieved a Dice score (a measure of overlap) within 1.4 to 1.9 points of a "fully supervised" model trained on hundreds of images.
  • Outperforming Baselines: The approach beat the next-best few-shot method (Contrastive Learning) by an average of 3.6 Dice points.
  • Robustness: Unlike standard FSL, which is highly sensitive to the specific images chosen for training, this method showed high stability regardless of which five images were selected for the initial labeling.
  • Scaling with Unlabeled Data: The performance gap between this method and baselines widened as more unlabeled data was added, proving the model effectively learns from the underlying structure of the unlabeled images.

Limitations and Future Work

While highly effective for larger organs like the liver or spleen, the model still struggles with very small or highly complex structures (such as the esophagus), where the performance gap compared to full supervision remains larger. Additionally, because all LFs are CNNs, they may share similar failure modes, potentially biasing the consensus.

Future work may explore "Active Learning" integrations, allowing the system to automatically suggest which five images would be the most informative for a human to label first, further optimizing the annotator's time.

Summary updated

Abstract

Constructing large, labeled training datasets for segmentation models is an expensive and labor-intensive process. This is a common challenge in machine learning, addressed by methods that require few or no labeled data points such as few-shot learning (FSL) and weakly-supervised learning (WS). Such techniques, however, have limitations when applied to image segmentation---FSL methods often produce noisy results and are strongly dependent on which few datapoints are labeled, while WS models struggle to fully exploit rich image information. We propose a framework that fuses FSL and WS for segmentation tasks, enabling users to train high-performing segmentation networks with very few hand-labeled training points. We use FSL models as weak sources in a WS framework, requiring a very small set of reference labeled images, and introduce a new WS model that focuses on key areas---areas with contention among noisy labels---of the image to fuse these weak sources. Empirically, we evaluate our proposed approach over seven well-motivated segmentation tasks. We show that our methods can achieve within 1.4 Dice points compared to fully supervised networks while only requiring five hand-labeled training points. Compared to existing FSL methods, our approach improves performance by a mean 3.6 Dice points over the next-best method.
BibTeX
@inproceedings{hooper2021cut,
  author = {Sarah M. Hooper and Michael Wornow and Ying Hang Seah and Peter Kellman and Hui Xue and Frederic Sala and Curtis P. Langlotz and Christopher Ré},
  title = {{Cut out the annotator, keep the cutout: better segmentation with weak supervision}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
}
Figure