RadioActive: 3D Radiological Interactive Segmentation Benchmark

Constantin Ulrich, Tassilo Wald, Emily Tempus, Maximilian Rokuss, Paul F. Jaeger, Klaus H. Maier-Hein

Introduction

In the fields of radiology and oncology, segmenting anatomical structures such as organs or tumors is a critical but labor-intensive task. While the "Segment Anything Model" (SAM) revolutionized interactive segmentation for natural images, its application to 3D medical data (CT and MRI) has been inconsistent. The RadioActive benchmark introduces a rigorous framework to bring order to this "Wild West" of 3D medical segmentation, establishing a standard for comparing 2D and 3D models fairly.

The Pitfalls of Current Research

The authors identify three major pitfalls in existing medical segmentation studies:

  1. Unrealistic Interaction (P1): Many benchmarks evaluate 2D models by assuming a clinician will provide prompts for every single slice of a 3D volume—a prohibitive time investment in clinical practice.
  2. Neglecting Refinement (P2): Most studies focus only on the "first click," ignoring the iterative process where humans correct AI errors.
  3. Inconsistent Evaluation (P3): A lack of standard datasets and metrics makes it impossible to determine which models truly lead the field.
The RadioActive benchmark overview, showcasing the diverse datasets, model pool, and evaluation pipeline.

Realistic Prompting Strategies

To address these pitfalls, RadioActive introduces Realistic Prompting Strategies that bridge the gap between 2D models and 3D volumes. These strategies simulate how a clinician would actually interact with a 3D dataset, allowing researchers to measure "Human Effort" (quantified by click count) on a level playing field.

  • Interpolation: Instead of prompting every slice, a user provides points or boxes for a few key slices (e.g., top, middle, and bottom). The system mathematically interpolates these prompts for all intermediate slices.
  • Propagation: The user prompts a single central slice, and the model uses its own prediction to "guess" the prompt location on adjacent slices, moving outward through the volume.
  • Scribble Refinement: To simulate real-world correction, an algorithm generates "scribbles" on areas where the model failed, testing how well a model learns from iterative feedback.
Detailed visualization of realistic prompting strategies including slice-wise prompting, interpolation, and propagation for both points and boxes.

Benchmark Methodology and Datasets

The benchmark utilizes 10 diverse, publicly available datasets covering CT and MRI modalities. To ensure a fair "zero-shot" evaluation, the authors selected datasets that were not part of the models' original training data. Target structures include healthy organs and various pathologies, such as MS lesions, lung tumors, and liver metastases.

Key Insights and Results

The benchmark yielded several surprising findings that challenge current assumptions in medical AI:

  • SAM 2 Dominance: General-purpose models are proving highly effective. SAM 2 emerged as the new state-of-the-art for medical imaging, achieving a mean Dice Similarity Coefficient (DSC) of 81.88 using box interpolation. It outperformed specialized models trained on millions of medical images.
  • The Power of Bounding Boxes: Bounding boxes consistently provide better spatial context than points. A few well-placed boxes are often sufficient for near-perfect segmentation.
  • Efficiency of 2D Models: RadioActive demonstrates that with smart interpolation, 2D models can match the efficiency of native 3D models without sacrificing accuracy.
  • Refinement is Essential: Models that lack support for iterative mask refinement (such as the original MedSAM) are significantly less useful in clinical settings where precision is paramount.

Among native 3D models, SegVol was the most competitive, particularly with point-based prompts, though it still struggled to match the box-based performance of SAM 2. However, the benchmark revealed that nearly all current models still struggle with small, low-contrast pathological lesions.

Conclusion and Future Work

RadioActive provides a necessary foundation for evaluating interactive 3D segmentation. While the current "Human Effort Proxy" (counting clicks) is a simplification, it offers a scalable way to compare model architectures. Future work will involve user studies with clinicians to measure real-world annotation time and cognitive load, as well as integrating the RadioActive codebase into standard medical imaging toolkits.

Summary updated

Abstract

Current interactive segmentation approaches, inspired by the success of META's Segment Anything model, have achieved notable advancements, however, they come with substantial limitations that hinder their practical application in 3D radiological scenarios. These include unrealistic human interaction requirements, such as slice-by-slice operations for 2D models on 3D data, a lack of iterative interactive refinement, and insufficient evaluation experiments. These shortcomings prevent accurate assessment of model performance and lead to inconsistent outcomes across studies. The RadioActive benchmark overcomes these challenges by offering a comprehensive and reproducible evaluation of interactive segmentation methods in realistic, clinically relevant scenarios. It includes diverse datasets, target structures, and interactive segmentation methods, and provides a flexible, extendable codebase that allows seamless integration of new models and prompting strategies. We also introduce advanced prompting techniques to enable 2D models on 3D data by reducing the needed number of interaction steps, enabling a fair comparison. We show that surprisingly the performance of slice-wise prompted approaches can match native 3D methods, despite the domain gap. Our findings challenge the current literature and highlight that models not specifically trained on medical data can outperform the current specialized medical methods. By open-sourcing RadioActive, we invite the research community to integrate their models and prompting techniques, ensuring continuous and transparent evaluation of interactive segmentation models in 3D medical imaging.
BibTeX
@misc{ulrich2024radioactive,
  author = {Constantin Ulrich and Tassilo Wald and Emily Tempus and Maximilian Rokuss and Paul F. Jaeger and Klaus H. Maier-Hein},
  title = {{RadioActive: 3D Radiological Interactive Segmentation Benchmark}},
  year = {2024},
  eprint = {2411.07885},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.07885},
}
Figure