BoxShrink: From Bounding Boxes to Segmentation Masks

Michael Gröger, Vadim Borisov, Gjergji Kasneci

TL;DR

BoxShrink is a novel, "training-free" framework that transforms simple bounding box annotations into detailed segmentation masks. By leveraging superpixels and conditional random fields (CRFs), it bridges the performance gap between weakly-supervised and fully-supervised models, offering a ~4% IoU improvement in medical imaging tasks without requiring expensive manual pixel-level labeling.

What Problem Does This Paper Address?

In the world of computer vision, especially medical imaging, data is gold, but labels are the bottleneck. While deep learning models (like CNNs) excel at image segmentation, they are "data-hungry."

The cost of labeling is hierarchical:

  1. Image Classification: Cheap and fast.
  2. Bounding Boxes: Relatively quick but imprecise.
  3. Semantic Segmentation: Extremely expensive and time-consuming, requiring experts (like doctors) to trace exact boundaries.

The authors address this "labeling bottleneck" by asking: Can we take the cheap bounding boxes we already have and automatically "shrink" them into accurate segmentation masks? This allows researchers to train high-quality segmentation models using only weak, low-cost supervision.

Methods

The authors propose BoxShrink, a framework that acts as a preprocessing step. Crucially, it does not require training a machine learning model to generate these masks. It comes in two flavors:

  1. rapid-BoxShrink: Designed for speed. It segments an image into superpixels (groups of pixels with similar colors/textures using the SLIC algorithm). It then keeps superpixels that overlap with the user-provided bounding box above a certain threshold. Finally, it applies a Fully-Connected Conditional Random Field (FCRF) to smooth the edges.
  2. robust-BoxShrink: Designed for precision. It starts with the "rapid" approach but adds a sophisticated refinement step. It uses a pretrained ResNet-50 to generate embeddings (numerical representations) for each superpixel. By comparing the "boundary" superpixels to the average foreground and background embeddings using cosine similarity, it iteratively decides whether to keep or discard ambiguous patches.

Datasets

The framework was evaluated on the CVC-Clinic DB, a standard dataset for medical image computing.

  • Content: 612 endoscopy frames showing polyps.
  • Labels: Originally contained binary ground truth masks. The authors inferred bounding boxes from these masks to simulate a weakly-supervised scenario.
  • Contribution: The authors have open-sourced their bounding box labels and code to help the community benchmark future weakly-supervised methods.

Key Insights and Ideas

  • Decoupled Preprocessing: Unlike many other methods, BoxShrink is a preprocessing step. It doesn't care what your final model architecture is (U-Net, DeepLab, etc.), making it "plug-and-play" for existing pipelines.
  • The Power of Superpixels: By moving from pixel-level decisions to superpixel-level decisions, the method becomes much more robust to noise and significantly faster.
  • No Training Required: Because it relies on classical algorithms (SLIC, CRFs) and pretrained embeddings, you don't need a massive GPU cluster just to generate your training labels.
  • Hardware Agnostic: The authors note that the label preprocessing doesn't require specialized hardware like GPUs or TPUs, making it accessible for smaller labs.

Key Results

The authors tested the generated masks by training several popular architectures (U-Net and DeepLabV3+) with various backbones (VGG-16, ResNet-18/50).

  • Accuracy Boost: Using BoxShrink-generated masks led to an average 4% improvement in Intersection over Union (IoU) compared to training models on raw bounding boxes.
  • Efficiency:
    • rapid-BoxShrink takes only ~0.5 seconds per image.
  • robust-BoxShrink takes ~3 seconds per image but generally yields higher precision.
  • Closing the Gap: The performance of models trained with BoxShrink masks approached the performance of models trained on "perfect" manually-labeled masks, significantly reducing the "weak supervision penalty."

Limitations and Future Work

  • Background Noise: In some "failure cases," the framework struggles to eliminate background pixels that look very similar to the foreground object (e.g., certain textures in colonoscopy images).
  • Binary Focus: Currently, the paper focuses on binary segmentation (object vs. background). The authors suggest that future work should explore multi-class problems.
  • Dataset Scale: While effective on the CVC-Clinic DB, the authors aim to validate the framework on much larger and more diverse medical datasets in the future.

Summary updated

Abstract

One of the core challenges facing the medical image computing community is fast and efficient data sample labeling. Obtaining fine-grained labels for segmentation is particularly demanding since it is expensive, time-consuming, and requires sophisticated tools. On the contrary, applying bounding boxes is fast and takes significantly less time than fine-grained labeling, but does not produce detailed results. In response, we propose a novel framework for weakly-supervised tasks with the rapid and robust transformation of bounding boxes into segmentation masks without training any machine learning model, coined BoxShrink. The proposed framework comes in two variants – rapid-BoxShrink for fast label transformations, and robust-BoxShrink for more precise label transformations. An average of four percent improvement in IoU is found across several models when being trained using BoxShrink in a weakly-supervised setting, compared to using only bounding box annotations as inputs on a colonoscopy image data set. We open-sourced the code for the proposed framework and published it online.
BibTeX
@inproceedings{groger2022boxshrink,
  author = {Michael Gröger and Vadim Borisov and Gjergji Kasneci},
  title = {{BoxShrink: From Bounding Boxes to Segmentation Masks}},
  booktitle = {MILLanD @ MICCAI},
  year = {2022},
  url = {https://openreview.net/forum?id=mnEn_GF3D_},
}

Loading related content...

Figure