TL;DR
Tyche is the first framework for stochastic in-context medical image segmentation, capable of generating multiple plausible masks for new, unseen tasks without any retraining or fine-tuning. By combining a novel "SetBlock" architecture with a "best candidate" loss, it captures the inherent ambiguity and expert disagreement in medical imaging while maintaining the flexibility of few-shot learning.
What Problem Does This Paper Address?
Medical image segmentation faces two major hurdles in clinical practice. First, most state-of-the-art models are task-specific; applying a model to a new modality or anatomy usually requires a machine learning expert to retrain or fine-tune it on a large labeled dataset. Second, standard models are deterministic, producing a single "correct" mask. In reality, medical images are often ambiguous due to low contrast or noise, and even expert radiologists frequently disagree on exact boundaries.
Existing "In-Context Learning" (ICL) models like UniverSeg solve the first problem by using a few example image-mask pairs (a context set) to define a task at inference time. However, they remain deterministic. Conversely, stochastic models like Probabilistic UNet capture uncertainty but must be trained from scratch for every new task. Tyche bridges this gap, providing a single model that is both task-agnostic and capable of expressing uncertainty through multiple diverse predictions.
Methods
The authors propose two variants: Tyche-TS (Train-time Stochasticity) and Tyche-IS (Inference-time Stochasticity).
Tyche-TS: The Core Architecture
Tyche-TS is a neural network that takes a target image, a context set, and a set of $K$ random noise channels as input. Its architecture is built around a modified UNet structure featuring a novel component called the SetBlock.
- SetBlock: Standard convolutions operate on single images. The SetBlock enables interactions between the entire context set and the set of $K$ prediction candidates. It uses a "CrossBlock" mechanism to compare image representations against the context set, allowing the model to "learn" the task definition on the fly.
- Best Candidate Loss: To ensure the $K$ predictions are diverse and don't just collapse into a single blurry average, the authors use a "min-loss" strategy. During training, the loss is only back-propagated through the single prediction that best matches the ground truth. This encourages different noise channels to specialize in different plausible segmentation styles.
Tyche-IS: The Test-Time Alternative
Tyche-IS is a simpler, training-free approach that applies In-Context Test-Time Augmentation (ICTTA) to existing deterministic ICL models. By applying random intensity transforms to both the target and the context set, it coaxes multiple different predictions out of a standard model.
Datasets
The authors evaluated Tyche on a massive scale:
- Training: A collection of 73 diverse biomedical datasets (MegaMedical) covering brain MRI, cardiac ultrasound, thoracic CT, and more.
- Multi-Annotator Evaluation: Datasets with labels from multiple experts, including LIDC-IDRI (lung nodules), Qubiq (brain, kidney, pancreas), and STARE (retinal vessels).
- Out-of-Distribution (O.D.) Testing: 20 tasks that were completely unseen during training to test the model's true generalization capabilities.
- Synthetic Data: The authors also used synthetic "blobs" with simulated annotator deformations to further boost the model's ability to handle diverse shapes.
Key Insights and Ideas
- Interaction is Key: The ablation studies showed that the SetBlock is the most critical architectural choice. Without it, the model's ability to produce diverse, high-quality candidates drops significantly.
- Implicit Characterization: Tyche-IS performs surprisingly well, suggesting that in-context models are naturally sensitive to small perturbations in the context set, which can be leveraged to model uncertainty without specialized training.
- Avoiding the "Mean" Trap: Traditional losses force models to predict the average of all annotators, which often results in a mask that no single expert would actually draw. The "best candidate" loss effectively bypasses this, allowing the model to explore the full space of plausible segmentations.
Key Results
- Performance: Tyche-TS consistently outperformed deterministic ICL benchmarks (like UniverSeg and SegGPT) and interactive methods (like SAM) on unseen tasks.
- Matching Specialists: Remarkably, Tyche matches the performance of specialized stochastic models (like Probabilistic UNet and PhiSeg) that were specifically trained on the test datasets, despite Tyche seeing those tasks for the first time at inference.
- Efficiency: Tyche-TS is highly efficient, generating 8 diverse predictions in just ~18ms on a V100 GPU, making it significantly faster than diffusion-based alternatives like CIDM.
- Generalization: As shown in Figure 6, Tyche produces plausible, diverse segmentations even for single-annotator data where the "correct" boundary is ambiguous.
Limitations and Future Work
While Tyche is a major step forward, the authors note a few limitations:
- Complex Structures: There is a performance dip on highly specific structures like individual vertebrae (SpineWeb), likely because the training data lacked similar examples.
- 2D Focus: The current implementation operates on 2D slices. Extending this to full 3D volumes while maintaining the "SetBlock" interaction efficiency is a clear next step.
- Annotator Modeling: Future versions could potentially take explicit annotator IDs or characteristics as part of the context to better steer the stochasticity.
Summary updated