BoxShrink: From Bounding Boxes to Segmentation Masks

Michael Gröger, Vadim Borisov, Gjergji Kasneci

Abstract

One of the core challenges facing the medical image computing community is fast and efficient data sample labeling. Obtaining fine-grained labels for segmentation is particularly demanding since it is expensive, time-consuming, and requires sophisticated tools. On the contrary, applying bounding boxes is fast and takes significantly less time than fine-grained labeling, but does not produce detailed results. In response, we propose a novel framework for weakly-supervised tasks with the rapid and robust transformation of bounding boxes into segmentation masks without training any machine learning model, coined BoxShrink. The proposed framework comes in two variants—rapid-BoxShrink for fast label transformations, and robust-BoxShrink for more precise label transformations. An average of four percent improvement in IoU is found across several models when being trained using BoxShrink in a weakly-supervised setting, compared to using only bounding box annotations as inputs on a colonoscopy image data set. We open-sourced the code for the proposed framework and published it online.

Introduction: The Labeling Bottleneck

In the domain of computer vision, and specifically within medical imaging, the availability of high-quality data is often overshadowed by the scarcity of precise labels. While deep learning architectures like Convolutional Neural Networks (CNNs) have demonstrated remarkable success in image segmentation, their performance is heavily dependent on large, annotated datasets.

The cost and effort associated with labeling follow a strict hierarchy:

  1. Image Classification: Low cost; requires only a categorical tag.
  2. Bounding Boxes: Moderate cost; requires identifying the spatial extent of an object.
  3. Semantic Segmentation: High cost; requires pixel-perfect boundaries, often necessitating the time of medical experts.

This "labeling bottleneck" limits the scalability of segmentation models. BoxShrink addresses this challenge by asking whether existing, low-cost bounding box annotations can be automatically refined into high-quality segmentation masks. By "shrinking" these boxes to fit the actual contours of the target objects, researchers can leverage weak supervision to achieve results that rival fully-supervised methods.

Methodology

BoxShrink is designed as a training-free preprocessing framework. Unlike many weakly-supervised methods that require an auxiliary model to be trained on the specific dataset, BoxShrink utilizes classical computer vision techniques and pretrained embeddings to generate masks. The framework is offered in two distinct versions:

1. rapid-BoxShrink

This variant is optimized for computational efficiency. It begins by segmenting the image into superpixels—clusters of pixels with similar color and texture—using the Simple Linear Iterative Clustering (SLIC) algorithm. The framework then evaluates the overlap between these superpixels and the provided bounding box. Superpixels that exceed a specific overlap threshold are retained. Finally, a Fully-Connected Conditional Random Field (FCRF) is applied to smooth the boundaries and ensure spatial consistency.

2. robust-BoxShrink

For applications requiring higher precision, robust-BoxShrink introduces a refinement layer. After the initial superpixel selection, the framework utilizes a pretrained ResNet-50 to generate numerical embeddings for each superpixel. By calculating the cosine similarity between "boundary" superpixels and the average embeddings of the foreground and background, the system iteratively decides whether to keep or discard ambiguous patches. This allows the framework to better distinguish the object from its immediate surroundings.

Experimental Setup

The framework was validated using the CVC-Clinic DB, a benchmark dataset containing 612 endoscopy frames of colonic polyps. To simulate a weakly-supervised environment, the authors derived bounding boxes from the original ground truth masks. This setup allowed for a direct comparison between models trained on raw bounding boxes, BoxShrink-refined masks, and the original manual segmentations.

Key Results and Insights

The effectiveness of BoxShrink was tested across multiple architectures, including U-Net and DeepLabV3+, using various backbones such as VGG-16 and ResNet-18/50.

  • Performance Gains: Models trained with BoxShrink-generated masks showed an average 4% improvement in Intersection over Union (IoU) compared to those trained on raw bounding boxes.
  • Computational Efficiency: The rapid variant processed images in approximately 0.5 seconds, while the robust variant took roughly 3 seconds. Both methods are significantly faster than manual pixel-level annotation.
  • Hardware Accessibility: Because the framework relies on classical algorithms and pretrained features rather than intensive training, it does not require specialized GPU or TPU clusters for label generation, making it accessible to smaller research labs.
  • Decoupled Architecture: As a preprocessing step, BoxShrink is "plug-and-play." It functions independently of the final segmentation model, allowing it to be integrated into any existing machine learning pipeline.

Limitations and Future Directions

While BoxShrink significantly narrows the gap between weak and full supervision, some challenges remain. The framework occasionally struggles with "background noise" when the surrounding texture is highly similar to the target object. Additionally, the current research focuses on binary segmentation (object vs. background). Future iterations of BoxShrink are expected to address multi-class segmentation problems and undergo validation on larger, more diverse clinical datasets.

Conclusion

BoxShrink provides a robust solution to the high cost of medical image annotation. By transforming cheap bounding boxes into accurate masks without the need for additional training, it offers a scalable path for developing high-performance segmentation models in data-constrained environments. The authors have open-sourced the code and the bounding box labels to facilitate further community benchmarking in weakly-supervised learning.

Summary updated

Abstract

One of the core challenges facing the medical image computing community is fast and efficient data sample labeling. Obtaining fine-grained labels for segmentation is particularly demanding since it is expensive, time-consuming, and requires sophisticated tools. On the contrary, applying bounding boxes is fast and takes significantly less time than fine-grained labeling, but does not produce detailed results. In response, we propose a novel framework for weakly-supervised tasks with the rapid and robust transformation of bounding boxes into segmentation masks without training any machine learning model, coined BoxShrink. The proposed framework comes in two variants - rapid-BoxShrink for fast label transformations, and robust-BoxShrink for more precise label transformations. An average of four percent improvement in IoU is found across several models when being trained using BoxShrink in a weakly-supervised setting, compared to using only bounding box annotations as inputs on a colonoscopy image data set. We open-sourced the code for the proposed framework and published it online.
BibTeX
@inproceedings{groger2022boxshrink,
  author = {Michael Gröger and Vadim Borisov and Gjergji Kasneci},
  title = {{BoxShrink: From Bounding Boxes to Segmentation Masks}},
  booktitle = {MILLanD @ MICCAI},
  year = {2022},
  url = {https://openreview.net/forum?id=mnEn_GF3D_},
}
Figure