TL;DR
The ReXGrounding Challenge marks the first large-scale effort to link free-text radiology findings to specific 3D regions in chest CT scans. The top-performing AI models now match inter-radiologist agreement levels for segmenting focal findings, though diffuse abnormalities and complex multi-lesion cases remain significant hurdles for the field.
What Problem Does This Paper Address?
Radiology reports are the primary way doctors communicate findings, yet they are almost entirely text-based. A report might mention a "5 mm nodule in the posterior segment of the right upper lobe," but it doesn't provide the exact coordinates on the 3D scan. This forces clinicians to manually hunt for findings during follow-up exams to measure changes over time.
"Visual grounding"—the task of automatically mapping these text descriptions to 3D voxels—is the missing link for creating verifiable medical AI. If an AI can't show you exactly where it sees a finding, its report isn't truly explainable. Before this challenge, existing models performed poorly on the descriptive, nuanced language used by radiologists, often failing to distinguish between similar-looking lesions in different anatomical locations.
Methods
The challenge organizers established a rigorous framework for evaluating 3D segmentation conditioned on free-text. The winning teams converged on a highly effective "recipe" for this task:
- Text Encoding: Most top teams used frozen large-scale text encoders (like Qwen3-Embedding) to convert complex sentences into high-dimensional vectors.
- Vision-Language Fusion: The most successful architecture was VoxTell, which uses a transformer decoder to allow text embeddings to "query" image features at the bottleneck of a 3D U-Net.
- Anatomical Context: Successful models used large input patches (up to $448 \times 448 \times 256$ voxels) to provide the network with enough anatomical landmarks to resolve location-based phrases like "lateral basal segment."
- Post-processing: Several teams used "anatomy-aware" filtering, where they restricted the final segmentation mask to the specific lung lobe or side mentioned in the text finding.
Datasets
The challenge utilized the ReXGroundingCT dataset, which is derived from the public CT-RATE dataset (scans from Istanbul Medipol University).
- Training Set: 2,992 scans with 7,687 findings. These used a "partial-instance" protocol, where only up to three representative instances of a finding were segmented.
- Validation/Test Sets: 500 scans (200 validation, 300 test) featuring 963 findings. Crucially, these were exhaustively annotated by board-certified radiologists, meaning every single instance of a described finding was segmented to create a "gold standard."
Key Insights and Ideas
- AI is Reaching Human-Level Agreement: The best model achieved a Dice score of 0.375. While this number might seem low in traditional segmentation tasks, it actually matches the agreement between two independent groups of human radiologists (who reached a Dice of 0.32 on the same data). This suggests we are approaching the ceiling of "ground truth" consistency.
- Focal vs. Non-Focal Divide: There is a massive performance gap based on the type of finding. Focal findings (like nodules or masses) are relatively well-handled. Non-focal findings (like diffuse emphysema or scarring) are significantly harder for both humans and AI to delineate consistently.
- Location vs. Morphology: Models are surprisingly good at identifying the type of abnormality but can be easily misled regarding its location. If a prompt is edited to swap "left" for "right," models often move the mask to the wrong side, even if no abnormality exists there, highlighting a reliance on text over visual evidence in ambiguous cases.
- Size Doesn't Matter (to the Model): Interestingly, removing or changing size descriptors (e.g., changing "5 mm" to "30 mm") had almost no effect on the models' segmentations, suggesting current architectures don't yet "understand" physical scale in the way radiologists do.
Key Results
- Top Performance: The winning team reached a Dice score of 0.375 and a hit rate of 0.77 (meaning they found at least a portion of the finding 77% of the time).
- Benchmark Leap: The challenge significantly advanced the state-of-the-art. The best pre-challenge model (VoxTell) scored 0.285 on the benchmark; the challenge winner improved this to 0.390 on the same subset.
- Qualitative Success: As seen in Figure 5, the top models can successfully ground complex findings across multiple lobes and handle detailed anatomical descriptions.
Limitations and Future Work
Despite the progress, the task is far from "solved." The Instance F1 score (which measures if the model found every lesion in a multi-lesion finding) remains low at 0.34, compared to 0.47 for humans.
Future research needs to address:
- Multi-center Generalization: All data came from a single hospital; performance on different scanners or populations is unknown.
- Non-Focal Definitions: We need better, standardized rules for how to segment diffuse diseases to reduce human annotator variability.
- Beyond the Lungs: The current work is limited to non-contrast chest CTs. Expanding to contrast-enhanced scans and other body regions (abdomen, head) is the next logical step.
- Grounded Generation: Moving from "grounding a given finding" to "generating a report where every sentence is automatically linked to the scan."
Summary updated