The Brain Tumor Segmentation (BraTS) Challenge: Local Synthesis of Healthy Brain Tissue via Inpainting
Florian Kofler, Felix Meissen, Felix Steinbauer, Robert Graf, Stefan K Ehrlich, Annika Reinke, Eva Oswald, Diana Waldmannstetter, Florian Hoelzl, Izabela Horvath, Oezguen Turgut, Suprosanna Shit, Christina Bukas, Kaiyuan Yang, Johannes C. Paetzold, Ezequiel de da Rosa, Isra Mekki, Shankeeth Vinayahalingam, Hasan Kassem, Juexin Zhang, Ke Chen, Ying Weng, Alicia Durrer, Philippe C. Cattin, Julia Wolleb, M. S. Sadique, M. M. Rahman, W. Farzana, A. Temtam, K. M. Iftekharuddin, Maruf Adewole, Syed Muhammad Anwar, Ujjwal Baid, Anastasia Janas, Anahita Fathi Kazerooni, Dominic LaBella, Hongwei Bran Li, Ahmed W Moawad, Gian-Marco Conte, Keyvan Farahani, James Eddy, Micah Sheller, Sarthak Pati, Alexandros Karagyris, Alejandro Aristizabal, Timothy Bergquist, Verena Chung, Russell Takeshi Shinohara, Farouk Dako, Walter Wiggins, Zachary Reitman, Chunhao Wang, Xinyang Liu, Zhifan Jiang, Elaine Johanson, Zeke Meier, Ariana Familiar, Christos Davatzikos, John Freymann, Justin Kirby, Michel Bilello, Hassan M Fathallah-Shaykh, Roland Wiest, Jan Kirschke, Rivka R Colen, Aikaterini Kotrotsou, Pamela Lamontagne, Daniel Marcus, Mikhail Milchenko, Arash Nazeri, Marc-André Weber, Abhishek Mahajan, Suyash Mohan, John Mongan, Christopher Hess, Soonmee Cha, Javier Villanueva-Meyer, Errol Colak, Priscila Crivellaro, Andras Jakab, Abiodun Fatade, Olubukola Omidiji, Rachel Akinola Lagos, O O Olatunji, Goldey Khanna, John Kirkpatrick, Michelle Alonso-Basanta, Arif Rashid, Miriam Bornhorst, Ali Nabavizadeh, Natasha Lepore, Joshua Palmer, Antonio Porras, Jake Albrecht, Udunna Anazodo, Mariam Aboian, Evan Calabrese, Jeffrey David Rudie, Marius George Linguraru, Juan Eugenio Iglesias, Koen Van Leemput, Spyridon Bakas, Benedikt Wiestler, Ivan Ezhov, Marie Piraud, Bjoern H Menze
The BraTS Inpainting Challenge introduces a novel task: synthetically "removing" brain tumors from MRI scans by filling the lesioned areas with realistic, healthy tissue. This approach allows standard neuroimaging algorithms designed exclusively for healthy brains to be reliably applied to pathological scans. The paper outlines the challenge framework, a novel healthy mask generation pipeline, and key findings from the inaugural 2023 competition.
What Problem Does This Paper Address?
A major bottleneck in clinical neuroimaging is that the vast majority of automated tools—such as those for brain anatomy parcellation, tissue segmentation, and brain extraction—are trained and optimized exclusively on healthy brains. When these algorithms encounter a brain with a structural anomaly like a tumor, they frequently fail or produce highly unreliable results.
To solve this dilemma, researchers need a way to virtually "erase" the pathology and replace it with a plausible reconstruction of healthy tissue. This task, known as medical image inpainting or local tissue synthesis, allows clinicians to run standard downstream analysis pipelines on tumor patients without modifying the underlying software.
Fig. 1: Challenge Task: Given a set of pathological and incomplete T1 MRI images with specified
(masked) voided areas, the participants are tasked to synthesize healthy versions of the pathological
brains. We cast the problem of synthesizing the healthy tissue as inpainting within the tumor area.
Methods
The challenge tasks participants with taking a T1-weighted MRI scan along with a binary mask of the "voided" (zeroed-out) pathological region and filling it with realistic healthy tissue contrasts.
Because true "ground truth" healthy tissue does not exist for a patient's tumor region, the organizers developed a clever strategy to train and evaluate these models using surrogate inpainting masks:
Mask Sampling: The algorithm extracts realistic tumor shapes from a pool of existing annotations.
Inverse Scaling: To ensure a realistic distribution, the pipeline pairs small surrogate masks with brains containing large actual tumors, and vice versa.
Positioning: The sampled mask undergoes random mirroring and rotation before being placed in a demonstrably healthy region of the brain, far away from the actual tumor.
During training, models learn to reconstruct these healthy regions. During testing, they are evaluated on their ability to infill these hidden, healthy surrogate zones.
Fig. 3: Healthy inpainting masks in the RSNA-ASNR-MICCAI BraTS 2023 inpainting chal-
lenge: Exemplary healthy mask generation procedure for one brain and one healthy mask. Panel A
shows the underlying T1 scan and panel B has the respective tumor annotation overlaid in red). T1
scan and tumor annotation are the input for our mask sampling algorithm (right side. The algorithm
randomly samples a mask, transforms it, and places it somewhere in the brain until a valid configuration
is found. Panel C shows a valid example of a healthy mask (green) in the same brain as the tumor.
These two different annotations are provided for the training set. The algorithms are supposed to inpaint
both the masked tumor as well as the masked healthy area.
Datasets
The challenge exclusively utilizes native T1-weighted MRI scans to minimize computational barriers and maximize generalizability across different clinical imaging protocols.
BraTS Glioma Dataset: Features 570 patient exams for the test set, narrowed down to 568 after rigorous manual curation by medical experts to ensure the surrogate masks were truly placed in healthy tissue.
BraTS Meningioma Dataset: Introduced for the 2024 iteration of the challenge, this dataset includes 284 patient exams (curated to 277 cases) to evaluate how well models generalize across different tumor pathologies and scanner types.
Key Insights and Ideas
The Dimensionality Dilemma: The competition revealed a stark trade-off between 2D and 3D modeling approaches. 2D models (such as 2D Diffusion) generated highly detailed, visually stunning axial slices but suffered from severe out-of-plane "stripe" artifacts when viewed from sagittal or coronal perspectives. Conversely, 3D models maintained spatial consistency across the entire volume but produced noticeably blurrier outputs.
The Metric vs. Perception Gap: Standard computer vision metrics like SSIM and PSNR do not perfectly align with clinical realism. A model can score highly by producing a smooth, blurry average of brain tissue, even though a medical expert would immediately recognize it as unnatural.
Key Results
In the 2023 challenge, the top-performing team (Ying-Weng et al.) utilized a 3D U-Net optimized with Mean Absolute Error (MAE) and SSIM losses. They achieved the best overall quantitative scores:
SSIM: 0.91
RMSE: 0.07
PSNR: 23.59 dB
The runner-up team (Domaso et al.) deployed a 2D Diffusion Model. While it ranked second quantitatively due to 3D inconsistencies, it produced the most visually pleasing and texturally realistic individual slices. Statistical bootstrapping confirmed that the top team rankings were exceptionally stable and robust against minor dataset perturbations.
Limitations and Future Work
The primary hurdle for widespread adoption is the high computational and financial cost of training high-fidelity 3D generative models (especially 3D diffusion models). Additionally, current architectures still struggle to perfectly balance high-frequency structural details with full 3D volumetric consistency.
To push the field forward, the organizers have expanded the 2024 challenge to include meningiomas to test cross-pathology robustness. They also advocate for developing "human surrogate models"—machine learning metrics trained to mimic expert human perception—to better evaluate the clinical viability of synthesized images.
Summary updated
Abstract
A myriad of algorithms for the automatic analysis of brain MR images is available to support clinicians in their decision-making. For brain tumor patients, the image acquisition time series typically starts with an already pathological scan. This poses problems, as many algorithms are designed to analyze healthy brains and provide no guarantee for images featuring lesions. Examples include, but are not limited to, algorithms for brain anatomy parcellation, tissue segmentation, and brain extraction. To solve this dilemma, we introduce the BraTS inpainting challenge. Here, the participants explore inpainting techniques to synthesize healthy brain scans from lesioned ones. The following manuscript contains the task formulation, dataset, and submission procedure. Later, it will be updated to summarize the findings of the challenge. The challenge is organized as part of the ASNR-BraTS MICCAI challenge.
BibTeX
@misc{kofler2023brain,
author = {Florian Kofler and Felix Meissen and Felix Steinbauer and Robert Graf and Stefan K Ehrlich and Annika Reinke and Eva Oswald and Diana Waldmannstetter and Florian Hoelzl and Izabela Horvath and Oezguen Turgut and Suprosanna Shit and Christina Bukas and Kaiyuan Yang and Johannes C. Paetzold and Ezequiel de da Rosa and Isra Mekki and Shankeeth Vinayahalingam and Hasan Kassem and Juexin Zhang and Ke Chen and Ying Weng and Alicia Durrer and Philippe C. Cattin and Julia Wolleb and M. S. Sadique and M. M. Rahman and W. Farzana and A. Temtam and K. M. Iftekharuddin and Maruf Adewole and Syed Muhammad Anwar and Ujjwal Baid and Anastasia Janas and Anahita Fathi Kazerooni and Dominic LaBella and Hongwei Bran Li and Ahmed W Moawad and Gian-Marco Conte and Keyvan Farahani and James Eddy and Micah Sheller and Sarthak Pati and Alexandros Karagyris and Alejandro Aristizabal and Timothy Bergquist and Verena Chung and Russell Takeshi Shinohara and Farouk Dako and Walter Wiggins and Zachary Reitman and Chunhao Wang and Xinyang Liu and Zhifan Jiang and Elaine Johanson and Zeke Meier and Ariana Familiar and Christos Davatzikos and John Freymann and Justin Kirby and Michel Bilello and Hassan M Fathallah-Shaykh and Roland Wiest and Jan Kirschke and Rivka R Colen and Aikaterini Kotrotsou and Pamela Lamontagne and Daniel Marcus and Mikhail Milchenko and Arash Nazeri and Marc-André Weber and Abhishek Mahajan and Suyash Mohan and John Mongan and Christopher Hess and Soonmee Cha and Javier Villanueva-Meyer and Errol Colak and Priscila Crivellaro and Andras Jakab and Abiodun Fatade and Olubukola Omidiji and Rachel Akinola Lagos and O O Olatunji and Goldey Khanna and John Kirkpatrick and Michelle Alonso-Basanta and Arif Rashid and Miriam Bornhorst and Ali Nabavizadeh and Natasha Lepore and Joshua Palmer and Antonio Porras and Jake Albrecht and Udunna Anazodo and Mariam Aboian and Evan Calabrese and Jeffrey David Rudie and Marius George Linguraru and Juan Eugenio Iglesias and Koen Van Leemput and Spyridon Bakas and Benedikt Wiestler and Ivan Ezhov and Marie Piraud and Bjoern H Menze},
title = {{The Brain Tumor Segmentation (BraTS) Challenge: Local Synthesis of Healthy Brain Tissue via Inpainting}},
year = {2023},
eprint = {2305.08992},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2305.08992},
}