Abstract
We compare two conditioning mechanisms based on concatenation and feature-wise modulation to integrate non-imaging information into convolutional neural networks for segmentation of anatomical structures. As a proof-of-concept, we provide the distribution of class labels obtained from ground truth masks to ensure strong correlation between the conditioning data and the segmentation maps. We evaluate the methods on the ACDC dataset and show that conditioning with non-imaging data improves the performance of segmentation networks. We observed that conditioning U-Net architectures was challenging, where no method gave significant improvement at full data scales. However, the same architecture without skip connections outperforms its baseline with feature-wise modulation, and the relative performance increases as the training size decreases.
Introduction
In clinical practice, radiologists rely on a holistic view of the patient, integrating imaging data with Electronic Health Record (EHR) information such as age, weight, and various biomarkers. Despite this, most deep learning models for medical image segmentation operate exclusively on pixel data. Integrating one-dimensional non-imaging vectors with three-dimensional imaging volumes—a process known as conditioning—remains an open research question in the context of pixel-perfect segmentation tasks. This paper explores how to effectively bridge this gap to boost performance, particularly in data-scarce environments.
Methodology
This study evaluates two primary conditioning strategies across two distinct architectures: a standard U-Net (which uses skip connections to preserve spatial detail) and a vanilla Encoder-Decoder (the same architecture but without skip connections).
- Concatenation-based Conditioning: Non-imaging data is appended to feature maps at the input (Early), bottleneck (Middle), or output (Late) stages. Both raw data and data processed through a Multi-Layer Perceptron (MLP) were tested.
- Feature-wise Linear Modulation (FiLM): This method uses non-imaging data to learn scaling and shifting factors applied to the network's feature maps, allowing the side information to modulate feature expression globally across the image.
The models were evaluated using the ACDC 2017 (Automated Cardiac Diagnosis Challenge) dataset, focusing on the segmentation of the myocardium and ventricular cavities. As a proxy for non-imaging biomarkers, the distribution of class labels was used to provide a strong conditioning signal.
Results and Analysis
Performance on Encoder-Decoder Architectures
For the Encoder-Decoder architecture, conditioning provided consistent and significant improvements. As shown in Table 1, the FiLM (Decoder) approach significantly outperformed the baseline across all data fractions. The relative improvement in Dice scores ranged from approximately 2% at full data to 19% in the most extreme low-data regime (1.5% of the dataset), where conditioning helped maintain model utility while the baseline struggled.
Performance on U-Net Architectures
In contrast, the U-Net architecture proved more resistant to conditioning at scale. While improvements were observed in low-data scenarios (e.g., at 6% data), the relative gains at 100% data were not statistically significant or consistent across methods (see Table 2). This suggests that the skip connections in U-Nets, which pass high-resolution spatial information directly from the encoder to the decoder, may allow the network to "ignore" the conditioning signal in favor of original image features.
Discussion
The results highlight a "Skip Connection Conflict": while skip connections are vital for high-quality segmentation, they appear to diminish the impact of external conditioning. However, conditioning acts as a powerful regularizer. The performance gap between conditioned models and baselines widened significantly as data became scarcer, suggesting that side information can act as a vital "hint" to compensate for a lack of labeled imaging data.
Conclusion
Conditioning convolutional networks with non-imaging data is a viable strategy for improving medical image segmentation, especially when training data is limited. While the U-Net remains a robust baseline, simpler architectures benefit immensely from feature-wise modulation (FiLM). Future work should explore hybrid architectures that can leverage both the spatial precision of skip connections and the integrative power of conditioning mechanisms for more robust clinical applications.
Summary updated