Classifying shoulder implants in X-ray images using deep learning

Gregor Urban, Saman Porhemmat, Maya Stark, Brian T. Feeley, Kazunori Okada, Pierre BaldiComputational 2020 · DOI

TL;DR

This paper introduces a deep learning approach to automatically classify shoulder implants in X-ray images, a crucial task for revision surgeries when implant details are unknown. They demonstrate that pre-trained deep convolutional neural networks (CNNs) significantly outperform traditional machine learning methods, achieving up to 80% accuracy in identifying implant manufacturers.

What Problem Does This Paper Address?

Total Shoulder Arthroplasty (TSA) is a common procedure to replace damaged shoulder joints with prostheses. Over time, these implants may require servicing or replacement. A significant challenge arises when patients, especially those who have moved or have incomplete medical records, cannot recall the manufacturer or model of their implant. This lack of information complicates revision surgeries, requiring careful visual inspection by experts to select the correct replacement equipment. This paper aims to automate this identification process using deep learning.

Methods

The researchers explored various deep learning models, primarily focusing on pre-trained Convolutional Neural Networks (CNNs) that were initially trained on the large ImageNet dataset. They utilized well-established architectures like VGG-16, VGG-19, ResNet-50, ResNet-152, DenseNet, and NASNet. For these pre-trained models, they removed the final fully connected layers (which are highly specific to ImageNet) and added a smaller, randomly initialized layer before fine-tuning them on their shoulder X-ray dataset. They also developed a custom CNN architecture from scratch for comparison.

To establish a baseline, they compared the deep learning models against traditional machine learning classifiers: Random Forests, Logistic Regression, Gradient Boosting, and K-Nearest Neighbors. All models, both deep learning and traditional, were trained with data augmentation techniques (random shifting, zooming, rotations, and flipping) to improve generalization.

Datasets

The study utilized a custom dataset comprising 597 de-identified X-ray scans of shoulder prostheses from four manufacturers (Cofield, Depuy, Tornier, and Zimmer) and a total of 16 different implant models. Some images were sourced from the University of Washington's shoulder website, while others came from individual surgeons and manufacturers. The dataset presented challenges such as variable image resolution, aspect ratios, contrast, and class imbalance.

Key Insights and Ideas

  • Pre-training is Crucial: The most significant finding is the dramatic performance improvement achieved by using CNNs pre-trained on ImageNet. This highlights the power of transfer learning, where knowledge gained from a general image recognition task can be effectively applied to a specialized medical imaging domain, even when the domains are quite different.
  • Fine-tuning the Entire Network is Better: When comparing fine-tuning the entire pre-trained CNN versus only fine-tuning the top hidden layers, the former yielded superior results. This suggests that adapting the deeper layers of the network is essential for capturing the specific features of shoulder implants.
  • Data Augmentation is Essential: Training CNNs without data augmentation led to a substantial drop in accuracy (around 14 percentage points), underscoring its importance for improving model robustness and generalization, especially with limited medical imaging data.
  • Deep Learning Outperforms Traditional Methods: CNNs, particularly when pre-trained, significantly outperformed traditional classifiers like Random Forests and Gradient Boosting, demonstrating the superiority of deep learning for this complex classification task.

Key Results

  • Top Performance: The NASNet model, pre-trained on ImageNet and fine-tuned, achieved the highest accuracy of approximately 80.4% in classifying implant manufacturers using 10-fold cross-validation.
  • Pre-trained CNNs vs. Others: Pre-trained CNNs achieved accuracies ranging from 74% to 80%, while non-pre-trained models (including custom CNNs and traditional classifiers) achieved accuracies between 51% and 56%. This difference was statistically significant.
  • Traditional Classifiers: Among the traditional classifiers, Random Forests performed best, reaching an accuracy of 56% with data augmentation.
  • Test-Time Augmentation: Augmenting test images at inference time did not significantly improve model performance, possibly due to hyperparameter settings being too extreme or the risk of overfitting on the limited dataset.

Limitations and Future Work

  • Class Imbalance: The dataset exhibits class imbalance, which is a common challenge in medical imaging. While the authors acknowledge this, they note that finding a universally optimal solution for imbalanced data is an ongoing research problem.
  • Out-of-Domain Manufacturers: The current model is trained on specific manufacturers. The paper suggests that future work should address how to handle test images from manufacturers not present in the training set, potentially by assessing model output confidence scores.
  • Model-Level Classification: The current work focuses on classifying manufacturers. The authors suggest that extending this to classify specific implant models would be a valuable next step, but this would require significantly more data.
  • Additional Manufacturers: Including a wider range of manufacturers in the dataset would further enhance the model's generalizability.

Summary updated

Abstract

Total Shoulder Arthroplasty (TSA) is a type of surgery in which the damaged ball of the shoulder is replaced with a prosthesis. Many years later, this prosthesis may be in need of servicing or replacement. In some situations, such as when the patient has changed his country of residence, the model and the manufacturer of the prosthesis may be unknown to the patient and primary doctor. Correct identification of the implant's model prior to surgery is required for selecting the correct equipment and procedure. We present a novel way to automatically classify shoulder implants in X-ray images. We employ deep learning models and compare their performance to alternative classifiers, such as random forests and gradient boosting. We find that deep convolutional neural networks outperform other classifiers significantly if and only if out-of-domain data such as ImageNet is used to pre-train the models. In a data set containing X-ray images of shoulder implants from 4 manufacturers and 16 different models, deep learning is able to identify the correct manufacturer with an accuracy of approximately 80% in 10-fold cross validation, while other classifiers achieve an accuracy of 56% or less. We believe that this approach will be a useful tool in clinical practice, and is likely applicable to other kinds of prostheses.
BibTeX
@inproceedings{urban2020classifying,
  author = {Gregor Urban and Saman Porhemmat and Maya Stark and Brian T. Feeley and Kazunori Okada and Pierre Baldi},
  title = {{Classifying shoulder implants in X-ray images using deep learning}},
  booktitle = {Computational},
  year = {2020},
  doi = {10.1016/j.csbj.2020.04.005},
}
Figure