Image- vs. histogram-based considerations in semantic segmentation of pulmonary hyperpolarized gas images

Nicholas J. Tustison, Talissa A. Altes, Kun Qing, Mu He, G. Wilson Miller, Brian Avants, Yun M. Shim, James C. Gee, John P. Mugler, Jaime F. MatamedRxiv 2021 · DOI

Abstract

Magnetic resonance imaging (MRI) using hyperpolarized gases has made possible the novel visualization of airspaces in the human lung, which has advanced research into the growth, development, and pathologies of the pulmonary system. In conjunction with the innovations associated with image acquisition, multiple image analysis strategies have been proposed and refined for the quantification of such lung imaging with much research effort devoted to semantic segmentation, or voxelwise classification, into clinically oriented categories based on ventilation levels. Given the functional nature of these images and the consequent sophistication of the segmentation task, many of these algorithmic approaches reduce the complex spatial image information to intensity-only considerations, which can be contextualized in terms of the intensity histogram. Although facilitating computational processing, this simplifying transformation results in the loss of important spatial cues for identifying salient image features, such as ventilation defects (a well-studied correlate of lung pathophysiology), as spatial objects. In this work, we discuss the interrelatedness of the most common approaches for histogram-based optimization of hyperpolarized gas lung imaging segmentation and demonstrate how certain assumptions lead to suboptimal performance, particularly in terms of measurement precision. In contrast, we illustrate how a convolutional neural network is optimized (i.e., trained) directly within the image domain to leverage spatial information. This image-based optimization mitigates the problematic issues associated with histogram-based approaches and suggests a preferred future research direction. Importantly, we provide the entire processing and evaluation framework, including the newly reported deep learning functionality, as open-source through the well-known Advanced Normalization Tools ecosystem.

The Challenge of Pulmonary MRI Segmentation

Hyperpolarized gas magnetic resonance imaging (MRI), utilizing isotopes like $^{129}$Xe or $^3$He, has revolutionized the visualization of pulmonary airspaces. This technology allows for the quantification of lung function through semantic segmentation—the process of labeling each voxel into categories such as "ventilation defect," "low ventilation," or "normal ventilation." However, traditional segmentation pipelines often rely heavily on 1D intensity histograms, a simplification that discards critical spatial information and introduces sensitivity to common MRI artifacts like noise and bias fields.

Methodology: From Histograms to Deep Learning

The researchers compared several established histogram-based algorithms against a modern image-based approach. Traditional methods include Linear Binning (which rescales histograms to a reference), K-means and Fuzzy C-means clustering, and GMM-MRF (Atropos), which incorporates limited spatial context via Markov Random Fields.

In contrast, the authors introduced "El Bicho," a deep learning model based on a 2D U-Net architecture with attention gating. Unlike histogram-centric methods, El Bicho is trained directly in the image domain, allowing it to learn anatomical shapes and textures. To evaluate these methods, the study utilized cohorts from the University of Virginia (UVa) and the Harvard Dataverse, alongside a robust data augmentation strategy designed to simulate MRI noise and nonlinear intensity distortions.

Simulated image distortions demonstrating how varying levels of noise (Max SD) and nonlinear intensity mapping affect the image histogram while the visual structure of the lung slice remains largely recognizable.

The "Histogram Trap"

A primary finding of this work is the "Histogram Trap." Histogram-based methods assume that pixel brightness is an absolute, stable metric. However, MRI scans are frequently affected by uneven brightness and noise that shift intensity values without altering the underlying anatomy. The study demonstrates that two images can maintain high structural similarity (appearing nearly identical to a human observer) while possessing radically different histograms. Algorithms that only "see" the histogram are easily fooled by these invisible shifts, leading to inconsistent clinical measurements.

Scatter plots showing the relationship between structural similarity (SSIM) and histogram correlation. The data reveals that while images remain structurally similar (high SSIM), their histograms can become highly decorrelated due to nonlinear distortions.

Results: Robustness and Precision

While all tested methods achieved high diagnostic accuracy (AUC > 0.80) in distinguishing healthy from diseased lungs, significant differences emerged in measurement precision. Under conditions of simulated noise and intensity warping, the CNN-based "El Bicho" demonstrated superior robustness. Using the Dice metric to measure segmentation overlap, the image-based approach maintained consistent performance where histogram-based methods faltered. Statistical analysis (Tukey HSD) confirmed that the deep learning approach provided a significant increase in robustness, mitigating the variance introduced by arbitrary reference cohorts or imaging artifacts.

Conclusion and Open Science

By shifting the optimization target from the 1D histogram to the 2D/3D image domain, researchers can leverage spatial context to produce more reliable clinical metrics. To support further research, the authors have integrated the "El Bicho" network and the entire evaluation framework into the open-source Advanced Normalization Tools (ANTsX) ecosystem. This move toward image-based optimization suggests a preferred future direction for the field, ensuring that functional lung imaging remains a precise tool for studying pulmonary pathologies.

Summary updated

Abstract

Abstract Magnetic resonance imaging (MRI) using hyperpolarized gases has made possible the novel visualization of airspaces in the human lung, which has advanced research into the growth, development, and pathologies of the pulmonary system. In conjunction with the innovations associated with image acquisition, multiple image analysis strategies have been proposed and refined for the quantification of such lung imaging with much research effort devoted to semantic segmentation, or voxelwise classification, into clinically oriented categories based on ventilation levels. Given the functional nature of these images and the consequent sophistication of the segmentation task, many of these algorithmic approaches reduce the complex spatial image information to intensity-only considerations, which can be contextualized in terms of the intensity histogram. Although facilitating computational processing, this simplifying transformation results in the loss of important spatial cues for identifying salient image features, such as ventilation defects (a well-studied correlate of lung pathophysiology), as spatial objects. In this work, we discuss the interrelatedness of the most common approaches for histogram-based optimization of hyperpolarized gas lung imaging segmentation and demonstrate how certain assumptions lead to suboptimal performance, particularly in terms of measurement precision. In contrast, we illustrate how a convolutional neural network is optimized (i.e., trained) directly within the image domain to leverage spatial information. This image-based optimization mitigates the problematic issues associated with histogram-based approaches and suggests a preferred future research direction. Importantly, we provide the entire processing and evaluation framework, including the newly reported deep learning functionality, as open-source through the well-known Advanced Normalization Tools ecosystem.
BibTeX
@misc{tustison2021image,
  author = {Nicholas J. Tustison and Talissa A. Altes and Kun Qing and Mu He and G. Wilson Miller and Brian Avants and Yun M. Shim and James C. Gee and John P. Mugler and Jaime F. Mata},
  title = {{Image- vs. histogram-based considerations in semantic segmentation of pulmonary hyperpolarized gas images}},
  year = {2021},
  doi = {10.1101/2021.03.04.21252588},
  howpublished = {medRxiv},
}
Figure