Grounding Radiology Reports in 3D Chest CT: Results of the ReXGrounding Challenge

Mohammed Baharoon, Mahmoud Hussain Alabbad, Sung Eun Kim, Xiaoman Zhang, Lucas Bijnens, Zirong Chen, Jiayu Wang, Xi Ouyang, Mingyang Yu, Weiqiu Jin, Xiaodan Ye, Dijia Wu, Kwang-Hyun Uhm, Inhwa Son, Rachel van 't Hull, Tim Veenboer, Cees Snoek, Kevin Groot Lipman, Daesung Kim, Junha Im, Youngjun Jun, Sangwon Lee, Katelyn Morrison, Adam Perer, Mijin Yun, Ali Piracha, Reem Alwasiah, Talal Yahyan, Abdulrahman Alamoudi, Mashael Alshehri, Rayan Alsager, Hannah Hancock, Anna M. Hunkemöller, Peter Youssef, Mona A. Alhammad, Afnan Aljumayan, Brecht Van Berkel, Eric Bijnens, Saud Alhussain, Maarten Steyvers, Rolf Symons, Bryan Eelen, Ayed Alaklubi, Maha Buraykan Alshammari, Amirah Fahad Alshammeri, Tomas Gilvydis, Ibrahim AlTurki, Monerah Almohideb, Ali Alburkani, Shahad Alhomaiani, Fatima Jabali, Maximilian Rokuss, Moritz Langenberg, Yannick Kirchhoff, Benjamin Hamm, Klaus Maier-Hein, Ibrahim Ethem Hamamci, Sezgin Er, Michael Moritz, Hassan R. Alomaish, Luyang Luo, Pranav RajpurkararXiv 2026 · arXiv

TL;DR

The ReXGrounding Challenge marks the first large-scale effort to link free-text radiology findings to specific 3D regions in chest CT scans. The top-performing AI models now match inter-radiologist agreement levels for segmenting focal findings, though diffuse abnormalities and complex multi-lesion cases remain significant hurdles for the field.

What Problem Does This Paper Address?

Radiology reports are the primary way doctors communicate findings, yet they are almost entirely text-based. A report might mention a "5 mm nodule in the posterior segment of the right upper lobe," but it doesn't provide the exact coordinates on the 3D scan. This forces clinicians to manually hunt for findings during follow-up exams to measure changes over time.

"Visual grounding"—the task of automatically mapping these text descriptions to 3D voxels—is the missing link for creating verifiable medical AI. If an AI can't show you exactly where it sees a finding, its report isn't truly explainable. Before this challenge, existing models performed poorly on the descriptive, nuanced language used by radiologists, often failing to distinguish between similar-looking lesions in different anatomical locations.

Methods

The challenge organizers established a rigorous framework for evaluating 3D segmentation conditioned on free-text. The winning teams converged on a highly effective "recipe" for this task:

  1. Text Encoding: Most top teams used frozen large-scale text encoders (like Qwen3-Embedding) to convert complex sentences into high-dimensional vectors.
  2. Vision-Language Fusion: The most successful architecture was VoxTell, which uses a transformer decoder to allow text embeddings to "query" image features at the bottleneck of a 3D U-Net.
  3. Anatomical Context: Successful models used large input patches (up to $448 \times 448 \times 256$ voxels) to provide the network with enough anatomical landmarks to resolve location-based phrases like "lateral basal segment."
  4. Post-processing: Several teams used "anatomy-aware" filtering, where they restricted the final segmentation mask to the specific lung lobe or side mentioned in the text finding.
Overview of the ReXGrounding task and challenge workflow.

Datasets

The challenge utilized the ReXGroundingCT dataset, which is derived from the public CT-RATE dataset (scans from Istanbul Medipol University).

  • Training Set: 2,992 scans with 7,687 findings. These used a "partial-instance" protocol, where only up to three representative instances of a finding were segmented.
  • Validation/Test Sets: 500 scans (200 validation, 300 test) featuring 963 findings. Crucially, these were exhaustively annotated by board-certified radiologists, meaning every single instance of a described finding was segmented to create a "gold standard."

Key Insights and Ideas

  • AI is Reaching Human-Level Agreement: The best model achieved a Dice score of 0.375. While this number might seem low in traditional segmentation tasks, it actually matches the agreement between two independent groups of human radiologists (who reached a Dice of 0.32 on the same data). This suggests we are approaching the ceiling of "ground truth" consistency.
  • Focal vs. Non-Focal Divide: There is a massive performance gap based on the type of finding. Focal findings (like nodules or masses) are relatively well-handled. Non-focal findings (like diffuse emphysema or scarring) are significantly harder for both humans and AI to delineate consistently.
  • Location vs. Morphology: Models are surprisingly good at identifying the type of abnormality but can be easily misled regarding its location. If a prompt is edited to swap "left" for "right," models often move the mask to the wrong side, even if no abnormality exists there, highlighting a reliance on text over visual evidence in ambiguous cases.
  • Size Doesn't Matter (to the Model): Interestingly, removing or changing size descriptors (e.g., changing "5 mm" to "30 mm") had almost no effect on the models' segmentations, suggesting current architectures don't yet "understand" physical scale in the way radiologists do.
Performance breakdown by finding category, showing the difficulty of non-focal findings.

Key Results

  • Top Performance: The winning team reached a Dice score of 0.375 and a hit rate of 0.77 (meaning they found at least a portion of the finding 77% of the time).
  • Benchmark Leap: The challenge significantly advanced the state-of-the-art. The best pre-challenge model (VoxTell) scored 0.285 on the benchmark; the challenge winner improved this to 0.390 on the same subset.
  • Qualitative Success: As seen in Figure 5, the top models can successfully ground complex findings across multiple lobes and handle detailed anatomical descriptions.
Examples of successful grounding for various findings.

Limitations and Future Work

Despite the progress, the task is far from "solved." The Instance F1 score (which measures if the model found every lesion in a multi-lesion finding) remains low at 0.34, compared to 0.47 for humans.

Future research needs to address:

  • Multi-center Generalization: All data came from a single hospital; performance on different scanners or populations is unknown.
  • Non-Focal Definitions: We need better, standardized rules for how to segment diffuse diseases to reduce human annotator variability.
  • Beyond the Lungs: The current work is limited to non-contrast chest CTs. Expanding to contrast-enhanced scans and other body regions (abdomen, head) is the next logical step.
  • Grounded Generation: Moving from "grounding a given finding" to "generating a report where every sentence is automatically linked to the scan."

Summary updated

Abstract

Segmenting each reported finding in a radiology scan would allow findings to be measured and tracked over time and let AI systems generate reports that are verifiable, but so far AI models have performed poorly at this task. To advance this task, we organized the ReXGrounding Challenge at MICCAI 2026, the first challenge dedicated to it. Given a non-contrast chest CT scan and the set of findings from its report, participants trained models to produce the corresponding set of 3D segmentations. The challenge used 2,992 training and 200 validation scans from the ReXGroundingCT dataset, and evaluated methods on a held-out test set of 300 scans with 582 findings, exhaustively annotated by board-certified radiologists. In total, 48 teams from 15 countries made 760 submissions, and the best method reached a Dice of 0.375 and a hit rate of 0.77, matching agreement between two independent radiologist groups (Dice 0.32, hit rate 0.76, on a 50-scan subset). On the 100 scans of the pre-challenge benchmark, the winner reached a Dice of 0.390, against 0.285 for VoxTell, the best model before the challenge, which five of the six best teams fine-tuned. Non-focal findings remained far harder to segment than focal ones, instance F1 stayed below inter-reader agreement (0.34 vs. 0.47), and models sometimes segmented the right abnormality in the wrong location. Top methods thus match radiologist agreement in Dice, but non-focal findings, findings with multiple instances, and adherence to the anatomical location described in the text remain open problems.
BibTeX
@misc{baharoon2026grounding,
  author = {Mohammed Baharoon and Mahmoud Hussain Alabbad and Sung Eun Kim and Xiaoman Zhang and Lucas Bijnens and Zirong Chen and Jiayu Wang and Xi Ouyang and Mingyang Yu and Weiqiu Jin and Xiaodan Ye and Dijia Wu and Kwang-Hyun Uhm and Inhwa Son and Rachel van 't Hull and Tim Veenboer and Cees Snoek and Kevin Groot Lipman and Daesung Kim and Junha Im and Youngjun Jun and Sangwon Lee and Katelyn Morrison and Adam Perer and Mijin Yun and Ali Piracha and Reem Alwasiah and Talal Yahyan and Abdulrahman Alamoudi and Mashael Alshehri and Rayan Alsager and Hannah Hancock and Anna M. Hunkemöller and Peter Youssef and Mona A. Alhammad and Afnan Aljumayan and Brecht Van Berkel and Eric Bijnens and Saud Alhussain and Maarten Steyvers and Rolf Symons and Bryan Eelen and Ayed Alaklubi and Maha Buraykan Alshammari and Amirah Fahad Alshammeri and Tomas Gilvydis and Ibrahim AlTurki and Monerah Almohideb and Ali Alburkani and Shahad Alhomaiani and Fatima Jabali and Maximilian Rokuss and Moritz Langenberg and Yannick Kirchhoff and Benjamin Hamm and Klaus Maier-Hein and Ibrahim Ethem Hamamci and Sezgin Er and Michael Moritz and Hassan R. Alomaish and Luyang Luo and Pranav Rajpurkar},
  title = {{Grounding Radiology Reports in 3D Chest CT: Results of the ReXGrounding Challenge}},
  year = {2026},
  eprint = {2611.23862},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23862},
}
Figure