Learning to segment anatomy and lesions from disparately labeled sources in brain MRI

Meva Himmetoglu, Ilja Ciernik, Ender Konukoglu

Abstract

Segmenting healthy tissue structures alongside lesions in brain Magnetic Resonance Images (MRI) remains a challenge due to lesion-caused disruption of anatomy and the lack of jointly labeled training datasets. We propose a method robust to these disruptions that can be trained from disparately labeled sets—without requiring jointly labeled samples. Our approach decouples healthy tissue and lesion segmentation into two paths, leveraging multi-sequence acquisitions and merging information via an attention mechanism. During inference, image-specific adaptation reduces the adverse influence of lesions on healthy tissue predictions. This adaptation is integrated into training through meta-learning and co-training. Our model demonstrates improved performance on anatomical structures and lesions in glioblastoma datasets compared to state-of-the-art methods.

Introduction

In clinical neuroimaging, the simultaneous segmentation of healthy brain anatomy and pathological lesions is critical for diagnosis, surgical planning, and longitudinal monitoring. However, two primary obstacles hinder the development of automated tools for this task. First, there is a significant labeling gap: most large-scale datasets provide either anatomical labels for healthy subjects (e.g., HCP, ADNI) or lesion labels for patients (e.g., BraTS), but rarely both for the same individual. Second, anatomical disruption caused by aggressive lesions like glioblastomas—which displace and distort surrounding tissues—often causes standard anatomical segmentation algorithms to fail when encountering "out-of-distribution" pathological appearances.

This paper introduces a deep learning framework designed to overcome these challenges by learning from disparately labeled sources and employing a meta-learning strategy for robust inference-time adaptation.

Methodology

The proposed framework is built upon three core pillars: a decoupled architecture, inference-time adaptation, and meta-learning.

Decoupled Two-Path Architecture

To prevent lesion features from "polluting" anatomical feature extraction, the model utilizes two separate U-Net branches. The anatomy branch processes T1-weighted (T1w) images, which offer high contrast for structural details. The lesion branch processes FLAIR images, which are highly sensitive to pathological changes. An attention mechanism then fuses these features, allowing the model to dynamically weigh information from both sequences based on the local context.

Inference-Time Adaptation

A key innovation of this work is the use of image-specific adaptation during testing. When the model encounters a new patient scan, it performs a brief optimization step. It identifies the lesion region and enforces a consistency loss: the anatomical predictions must remain stable even if the internal texture of the lesion area is randomized. This forces the anatomy branch to rely on global context and ignore the confusing signals from within the tumor.

Meta Co-Training

To ensure the model can adapt effectively at test time, the authors employ a meta-learning strategy during training. They synthesize "pseudo-unhealthy" images by transplanting lesions into healthy scans. The model is trained to minimize an "outer loss" (segmentation accuracy) after it has taken an "inner step" (the consistency adaptation). This "learning to learn" approach ensures that the model parameters are optimized for rapid, effective adaptation to real-world lesions.

Experimental Results

The model was evaluated against established baselines, including SAMSEG, SynthSeg, and jSTABL, using the BraTS 2018 dataset and an in-house longitudinal glioblastoma dataset.

Quantitative Performance

The proposed method achieved state-of-the-art results on the BraTS test set. Specifically, it reached a Dice score of 0.882 for gray matter and 0.921 for white matter. This significantly outperformed the previous state-of-the-art, jSTABL, which achieved 0.840 and 0.883 respectively. These results demonstrate that the model maintains high anatomical accuracy even in the presence of significant pathology.

Qualitative Robustness and Generalization

Qualitative analysis showed that the model successfully delineates complex borders, such as the ventricles and basal ganglia, even when they are compressed by large tumors. Furthermore, testing on an in-house longitudinal dataset from Städtische Klinikum Dessau (48 patients) confirmed that the model provides consistent segmentations across pre- and post-operative scans, a vital requirement for tracking disease progression.

Discussion and Limitations

While the framework offers superior accuracy, it introduces a higher computational cost due to the optimization steps required at inference time, taking several minutes per volume compared to seconds for standard feed-forward models. Additionally, the memory intensity of calculating second-order gradients during meta-learning necessitated training on 3D patches rather than full volumes. Future work could explore handling missing modalities or incorporating more diverse sequences like T1-contrast or T2-weighted images.

Conclusion

By decoupling anatomical and lesion segmentation and utilizing meta-learning for inference-time adaptation, this framework successfully bridges the gap between disparately labeled datasets. It provides a robust solution for joint brain segmentation, maintaining high anatomical fidelity even in the presence of severe pathological disruptions.

Summary updated

Abstract

Segmenting healthy tissue structures alongside lesions in brain Magnetic Resonance Images (MRI) remains a challenge for today's algorithms due to lesion-caused disruption of the anatomy and lack of jointly labeled training datasets, where both healthy tissues and lesions are labeled on the same images. In this paper, we propose a method that is robust to lesion-caused disruptions and can be trained from disparately labeled training sets, i.e., without requiring jointly labeled samples, to automatically segment both. In contrast to prior work, we decouple healthy tissue and lesion segmentation in two paths to leverage multi-sequence acquisitions and merge information with an attention mechanism. During inference, an image-specific adaptation reduces adverse influences of lesion regions on healthy tissue predictions. During training, the adaptation is taken into account through meta-learning and co-training is used to learn from disparately labeled training images. Our model shows an improved performance on several anatomical structures and lesions on a publicly available brain glioblastoma dataset compared to the state-of-the-art segmentation methods.
BibTeX
@article{himmetoglu2025learning,
  author = {Meva Himmetoglu and Ilja Ciernik and Ender Konukoglu},
  title = {{Learning to segment anatomy and lesions from disparately labeled sources in brain MRI}},
  journal = {Medical Image Analysis},
  year = {2025},
  doi = {10.1016/j.media.2025.103705},
}
Figure