Automated Skeletal Bone Age Assessment with Two-Stage Convolutional Transformer Network Based on X-ray Images

Xiongwei Mao, Qinglei Hui, Siyu Zhu, Wending Du, Chenhui Qiu, Xiaoping Ouyang, Dexing KongDiagnostics 2023 · DOI

TL;DR

This paper introduces a novel two-stage convolutional transformer network for automated bone age assessment (BAA) from X-ray images. By mimicking radiologists' focus on specific bone regions (ROIs) and incorporating biological sex as a feature, their method achieves state-of-the-art accuracy, significantly reducing clinical workload and improving diagnostic speed and precision.

What Problem Does This Paper Address?

Bone age assessment (BAA) is crucial for understanding a child's growth and development, aiding in the diagnosis of endocrine disorders and even in legal contexts. However, traditional BAA methods, like the Greulich-Pyle (GP) atlas and Tanner-Whitehouse (TW) scores, are time-consuming, subjective, and prone to inter-observer variability. While deep learning has shown promise, many existing methods focus on global image features, neglecting the specific ossification regions that radiologists meticulously examine. This paper aims to bridge this gap by developing an automated system that replicates the clinical BAA process more closely, focusing on key bone regions and leveraging advanced deep learning architectures for improved accuracy and efficiency.

Methods

The proposed method employs a two-stage approach:

  • Stage 1: Region of Interest (ROI) Detection and Alignment:
    • Object Detection (YOLOv5): To mimic a radiologist's focus, the system first uses YOLOv5 to detect and extract 18 specific Regions of Interest (ROIs) corresponding to key joints and bones identified in the TW3 guidelines. This is a departure from methods that analyze the entire hand image.
  • Hand Bone Posture Alignment: To mitigate variations caused by X-ray imaging angles and patient positioning (e.g., mirroring, flipping, rotation), a novel alignment procedure is introduced. This step standardizes the hand bone posture, reducing noise and improving feature extraction.
  • Sex Token Integration: Biological sex is a known factor influencing bone development. Instead of treating it as a separate input or training separate models, the authors encode sex information into a "sex token" and fuse it with the ROI features. This is a novel approach to incorporating prior knowledge into the transformer's attention mechanism.
  • Stage 2: Swin Transformer Bone Age Prediction:
    • Swin Transformer Architecture: The aligned ROI features, augmented with the sex token, are fed into a Swin Transformer network. This architecture, known for its efficiency in computer vision tasks, uses shifted windows (SW-MSA) and windowed multi-head self-attention (W-MSA) to capture both local and cross-region feature interactions. This allows the model to learn complex relationships between different bone ossification stages.
  • Hybrid Loss Function: A custom hybrid loss function is developed, combining three components:
  • Cross-entropy loss: For classification-based learning.
  • Mean loss: Penalizes the difference between the predicted mean age distribution and the true age, focusing on regression accuracy.
  • Variance loss: Penalizes the dispersion of the predicted age distribution, promoting stable and confident predictions. This hybrid approach aims to balance accuracy and robustness.

Datasets

The study utilizes the 2017 Pediatric Bone Age Challenge dataset organized by the Radiological Society of North America (RSNA). This dataset comprises:

  • 14,236 X-ray images in total.
  • 12,611 images for training.
  • 1425 images for validation.
  • 200 images for testing.

The authors performed data cleaning, removing images with fewer than 14 detected ROIs or those exhibiting dysplasia or lesions, resulting in 12,574 training samples.

Key Insights and Ideas

  • Mimicking Radiologists' Focus: The core idea of using object detection to extract specific ROIs, rather than processing the entire hand image, is a significant step towards replicating the clinical BAA process. This highlights that not all parts of the image are equally informative for BAA.
  • Hand Bone Posture Alignment: Addressing the variability in X-ray imaging through automated alignment is crucial for robust feature extraction and reduces the need for manual preprocessing or reliance on models that are overly sensitive to pose variations.
  • Sex as a "Token": Integrating biological sex as a "sex token" within the transformer's attention mechanism is an innovative way to inject prior knowledge, proving more effective than separate training by sex. This demonstrates the power of feature fusion in deep learning models.
  • Hybrid Loss for Stability: The combination of mean, variance, and cross-entropy losses is a sophisticated approach to simultaneously optimize for accurate age prediction and robust, well-localized age distributions, addressing potential fluctuations during training.

Key Results

The proposed two-stage convolutional transformer network achieved impressive results:

  • Mean Absolute Error (MAE):
    • Validation Set: 6.22 months
    • Testing Set: 4.586 months
  • Cumulative Accuracy:
    • Within 6 months: 71% (testing set)
    • Within 12 months: 96% (testing set)
  • Ablation Studies: The results clearly show that using ROIs combined with the sex token yielded the best performance, outperforming models that used the whole image or trained separately by sex.
  • Comparison: The MAE of 4.586 months on the testing set is comparable to state-of-the-art methods, demonstrating the effectiveness of their approach.

Limitations and Future Work

  • Dataset Imbalance: The study acknowledges that the dataset had an unbalanced distribution of images across different age groups, which may affect the generalization of the algorithm.
  • Image Quality: The presence of dark X-ray images with low contrast could have impacted feature learning.
  • Global Information Neglect: While focusing on specific ROIs is beneficial, the method might overlook some global contextual information present in the entire image, which could be relevant.

Future work suggested by the authors includes:

  • Collecting additional data and balancing image distributions to improve generalization.
  • Exploring advanced data preprocessing techniques.
  • Investigating attention mechanisms for global-local contextual information fusion, potentially combining GP atlas and TW scoring methods.
  • Exploring multimodal data fusion (X-ray, ultrasound, MRI) for BAA.
  • Developing methods for generating intuitive explanations for radiologists to aid clinical diagnosis.

Summary updated

Abstract

Human skeletal development is continuous and staged, and different stages have various morphological characteristics. Therefore, bone age assessment (BAA) can accurately reflect the individual's growth and development level and maturity. Clinical BAA is time consuming, highly subjective, and lacks consistency. Deep learning has made considerable progress in BAA in recent years by effectively extracting deep features. Most studies use neural networks to extract global information from input images. However, clinical radiologists are highly concerned about the ossification degree in some specific regions of the hand bones. This paper proposes a two-stage convolutional transformer network to improve the accuracy of BAA. Combined with object detection and transformer, the first stage mimics the bone age reading process of the pediatrician, extracts the hand bone region of interest (ROI) in real time using YOLOv5, and proposes hand bone posture alignment. In addition, the previous information encoding of biological sex is integrated into the feature map to replace the position token in the transformer. The second stage extracts features within the ROI by window attention, interacts between different ROIs by shifting the window attention to extract hidden feature information, and penalizes the evaluation results using a hybrid loss function to ensure its stability and accuracy. The proposed method is evaluated on the data from the Pediatric Bone Age Challenge organized by the Radiological Society of North America (RSNA). The experimental results show that the proposed method achieves a mean absolute error (MAE) of 6.22 and 4.585 months on the validation and testing sets, respectively, and the cumulative accuracy within 6 and 12 months reach 71% and 96%, respectively, which is comparable to the state of the art, markedly reducing the clinical workload and realizing rapid, automatic, and high-precision assessment.
BibTeX
@inproceedings{mao2023automated,
  author = {Xiongwei Mao and Qinglei Hui and Siyu Zhu and Wending Du and Chenhui Qiu and Xiaoping Ouyang and Dexing Kong},
  title = {{Automated Skeletal Bone Age Assessment with Two-Stage Convolutional Transformer Network Based on X-ray Images}},
  booktitle = {Diagnostics},
  year = {2023},
  doi = {10.3390/diagnostics13111837},
}
Figure