Intelligent Evaluation Method for Scoliosis at Home Using Back Photos Captured by Mobile Phones

Yongsheng Li, Xin Peng, Qing Mao, Minglin Ma, Jiaqi Huang, Shuo Zhang, Dong Shao-jie, Zhihui Zhou, Yue Lan, Yu Pan, Ruimou Xie, Peiwu Qin, Kehong YuanBioengineering 2024 · DOI

TL;DR

This paper introduces a novel, multi-dimensional deep learning approach for evaluating scoliosis using only mobile phone photos of a patient's back. It aims to overcome the limitations of traditional X-ray examinations by enabling convenient, at-home screening and dynamic monitoring, offering a more accessible and radiation-free alternative.

What Problem Does This Paper Address?

Scoliosis, a spinal curvature, is increasingly common in adolescents. Early detection and intervention are crucial to prevent severe deformities. Current gold-standard diagnostic methods, like X-rays, are not ideal for large-scale screening or continuous monitoring due to radiation exposure, cost, and inflexibility. While advanced imaging techniques exist, they often require professional equipment and are not suitable for home use. This paper addresses the need for a non-invasive, accessible, and accurate method for evaluating scoliosis severity and type using readily available mobile phone cameras, facilitating both initial screening and long-term rehabilitation tracking.

Methods

The proposed method integrates three key deep learning-based algorithms:

  • Spinal Coronal Curvature Type Discrimination: This algorithm leverages YOLOv8 for keypoint detection on back images. It then uses curve fitting (specifically, least squares fitting) to model the spine's shape and identifies "C", "S", or multi-curve types based on the number of tangent points detected using a sliding window approach.
  • Coronal Plane Deviation Evaluation: This involves two sub-methods:
  • Cobb Angle Measurement: Based on the fitted spine curve, this method calculates the Cobb angle, the standard measure of scoliosis severity. It adapts the calculation for "C", "S", and multi-curve types.
  • "Scapular Spine Triangle" Difference: This method forms a triangle using the two scapular points and the spine's endpoint. The absolute difference between the angles at the base of this triangle is used to quantify coronal plane bending.
  • Trunk Rotation (ATR Angle) Measurement: This algorithm focuses on the sagittal plane. It involves:
  • Salient Object Segmentation: Using SAM (Segment Anything Model) to isolate the human back from the background.
  • Contour Extraction: Employing edge detection operators (Roberts, Prewitt, LOG) to outline the back. Prewitt is found to be most suitable.
  • Trunk Rotation Calculation: Utilizing the Automatic Multi-scale Peak Detection (AMPD) algorithm to identify key points (peaks) on the back contour, specifically the protruding scapular points. The angle between these points and the horizontal line is then calculated to determine trunk rotation.

A fusion method is also proposed, combining the "Scapular Spine Triangle" difference for "C" type scoliosis and the Cobb angle measurement for "S" and multi-curve types to maximize accuracy.

Datasets

  • Training and Testing: A dataset of 3640 standing half-length back photos, manually annotated by professional doctors, was used. This dataset was partitioned into training, verification, and test sets.
  • Clinical Matched Data: 25 patient back photos paired with their corresponding X-rays were collected from a hospital.
  • Trunk Rotation Dataset: 50 high-quality 90-degree backward bending photos were used.

Key Insights and Ideas

  • Leveraging Mobile Photos: The core innovation is demonstrating that high-quality scoliosis assessment can be achieved using readily available mobile phone cameras, democratizing access to screening.
  • Multi-Dimensional Approach: Moving beyond just coronal plane assessment, the paper incorporates sagittal plane (trunk rotation) evaluation, offering a more comprehensive view of spinal deformity.
  • Integration of Clinical Knowledge: The methods, particularly the "Scapular Spine Triangle" and the fusion approach, explicitly incorporate established clinical observations and doctor expertise, enhancing interpretability and reliability.
  • YOLOv8 for Keypoint Detection: The successful application of YOLOv8 for precise keypoint detection on back images is a foundational element for subsequent analyses.
  • AMPD for Peak Detection: The use of AMPD for robust peak detection on back contours is crucial for accurate trunk rotation measurement, especially in the presence of noise.

Key Results

  • Coronal Curvature Type Discrimination: Achieved an accuracy rate of 88.7%.
  • Cobb Angle Measurement: Showed improved accuracy compared to existing deep learning and convolutional neural network methods, with a reported 17.0% improvement over one method and 7.6% over another.
  • "Scapular Spine Triangle" Method: Demonstrated good performance, with accuracy improvements of 18.4% and 9.0% over comparable methods. However, it showed instability across different scoliosis types.
  • Fusion Method: Achieved the highest accuracy (0.793) among the evaluated methods, outperforming others by a significant margin.
  • Clinical Matched Data Verification: The proposed algorithm achieved an accuracy of 0.702 on a small set of clinically matched data.
  • Trunk Rotation Measurement: The Prewitt operator for contour extraction resulted in the lowest average relative error (6.2%) for trunk rotation measurement, indicating high accuracy.

Limitations and Future Work

  • Dataset Size and Diversity: While 3640 images were used, larger and more diverse datasets could further improve model generalization. The clinical matched data set was also small (25 patients).
  • "Scapular Spine Triangle" Instability: The authors note that the "Scapular Spine Triangle" method's performance is unstable across different scoliosis types ("S" and multi-curve).
  • Future Work: The authors envision deploying this multi-dimensional assessment algorithm into an application for easy patient use. They also plan to develop a remote monitoring and management system for continuous patient-doctor communication and guided rehabilitation, leveraging telemedicine.

Summary updated

Abstract

The traditional scoliosis examination based on X-ray film is not suitable for large-scale screening, and it is also not suitable for dynamic evaluation during rehabilitation. Therefore, based on computer vision technology, this paper puts forward an evaluation method of scoliosis with different photos of the back taken by mobile phones, which involves three aspects: first, based on the key point detection model of YOLOv8, an algorithm for judging the type of spinal coronal curvature is proposed; second, an algorithm for evaluating the coronal plane of the spine based on the key points of the human back is proposed, aiming at quantifying the deviation degree of the spine in the coronal plane; third, the measurement algorithm of trunk rotation (ATR angle) based on multi-scale automatic peak detection (AMPD) is proposed, aiming at quantifying the deviation degree of the spine in sagittal plane. The public dataset and clinical paired data (mobile phone photo and X-ray) are used to test. The results show that this method has high accuracy and effectiveness in distinguishing the type of spinal curvature and evaluating the degree of deviation, which is higher than other deep learning algorithms.
BibTeX
@inproceedings{li2024intelligent,
  author = {Yongsheng Li and Xin Peng and Qing Mao and Minglin Ma and Jiaqi Huang and Shuo Zhang and Dong Shao-jie and Zhihui Zhou and Yue Lan and Yu Pan and Ruimou Xie and Peiwu Qin and Kehong Yuan},
  title = {{Intelligent Evaluation Method for Scoliosis at Home Using Back Photos Captured by Mobile Phones}},
  booktitle = {Bioengineering},
  year = {2024},
  doi = {10.3390/bioengineering11111162},
}
Figure