Machine Learning Model as a Useful Tool for Prediction of Thyroid Nodules Histology, Aggressiveness and Treatment-Related Complications

Valeria Dell’Era, Alan Perotti, Michele Starnini, Massimo Campagnoli, Maria Silvia Rosa, Irene Saino, Paolo Aluffi Valletti, Massimiliano GarzaroJournal of Personalized Medicine 2023 · DOI

TL;DR

This paper introduces a machine learning (ML) model designed to predict thyroid nodule characteristics like histology, aggressiveness, and post-surgical complications. By leveraging a comprehensive dataset of patient information, the study demonstrates that ML can accurately stratify risk and potentially improve diagnostic tools for thyroid cancer, offering a valuable adjunct to traditional methods.

What Problem Does This Paper Address?

Thyroid nodules are extremely common, with a significant minority being malignant. While well-differentiated thyroid cancer generally has a good prognosis, some aggressive variants can be challenging to identify early. Current diagnostic tools like ultrasound and fine-needle aspiration biopsy (FNAB) are effective but have limitations, including potential sampling errors and interpretative challenges, leading to misdiagnoses and over-diagnosis. The paper aims to address the need for more accurate and refined diagnostic tools by exploring the application of machine learning to predict crucial aspects of thyroid nodules, including their malignancy, aggressiveness, and potential for post-surgical complications.

Methods

The researchers employed a retrospective study design, collecting data from 1218 patients who underwent thyroid surgery between 2006 and 2018. Their ML pipeline involved several key steps:

  • Data Cleaning and Preprocessing: Features with over a third of missing values were discarded. For remaining numerical features, missing data was imputed using the median. Categorical data with missing values was assigned "Unknown." To address data sparsity, binary YES/NO variables were created from multiclass categorical or numerical features.
  • Feature Engineering: A crucial derived feature, "DIAGNOSTIC PRE," was created by combining ultrasound (US) data, FNAB results, and clinical presentation, categorized according to American Thyroid Association (ATA) guidelines to indicate suspicion for malignancy. Another derived feature, "AGGRESSIVENESS YES/NO," was based on ATA criteria for aggressive pathological subtypes, extra-thyroidal extension, lymph node involvement, and distant metastasis.
  • Model Training: The dataset was split into training (75%) and testing (25%) sets. To handle imbalanced classes (common in complication prediction), the Synthetic Minority Oversampling Technique (SMOTE) was applied to the training data after the train-test split to prevent data leakage. Three off-the-shelf classifiers (Random Forest, Multilayer Perceptron, and k-Nearest Neighbors) were initially compared, with a focus on Random Forest for its explainability. The Random Forest model was trained using 3-fold cross-validation, with hyperparameter tuning performed via cross-validated grid search.
  • Prediction Tasks: The ML models were trained to predict two main events: (i) tumor histology and aggressiveness (including T and N staging), and (ii) complications (transient hypocalcemia and post-operative recovery duration). Predictions were made using all available features and then exclusively using pre-surgery features to assess the impact of temporal data.
  • Explainability: Feature importance was analyzed using Shapley Additive Explanations (SHAP) values to understand which factors most influenced the model's predictions.

Datasets

The study utilized a retrospective dataset comprising data from 1218 patients who underwent thyroid surgery at the ENT Division of Eastern Piedmont University, Novara, Italy, between January 2006 and December 2018. This dataset was curated to include both pre- and post-surgery features. The paper highlights the creation of this "rich, hand-curated, open dataset" as one of its main contributions.

Key Insights and Ideas

  • Pre-screening is King: The study strongly suggests that pre-operative features, particularly those related to initial screening (like FNAB and ultrasound findings), are highly predictive of tumor histology and aggressiveness. This implies that a thorough initial assessment can significantly inform the risk stratification of thyroid nodules.
  • fT3 as an Aggressiveness Indicator: In this specific population, an out-of-range (low) fT3 dosage during the pre-operative examination was found to be strongly associated with a higher aggressiveness of the disease. This is a notable finding that could guide clinical attention.
  • Explainable AI for Clinical Trust: The researchers focused on developing explainable ML models (specifically Random Forest with SHAP values). This is crucial for clinical adoption, as it allows clinicians to understand why a model makes a certain prediction, fostering trust and enabling better clinical decision-making.
  • Post-Surgery Data's Limited Predictive Power for Complications: Interestingly, the study found that post-surgery features did not significantly improve the prediction accuracy for transient hypocalcemia and other complications. This suggests that pre-operative factors are largely sufficient for predicting these outcomes.

Key Results

  • High Prediction Accuracy: The ML models achieved high accuracy across various prediction tasks. For instance, predicting tumor histology and aggressiveness was accurate at over 90%, with only a small drop (4-5%) when using only pre-surgery features. T/N prediction was over 95% accurate with all features, dropping to around 84-88% with pre-surgery data alone.
  • Complication Prediction Robustness: Transient hypocalcemia and general complications were predicted with nearly 80-84% accuracy, showing no significant drop when using only pre-surgery features.
  • Recovery Duration Prediction: Post-surgery recovery duration was predicted with over 90% accuracy using all features, with a ~10% drop when relying solely on pre-surgery data.
  • Feature Importance: SHAP analysis revealed that features like "DIAG_PRE" (clinic, FNAC, and US suggested malignancy) and "CIT_PRE_YN" (FNAC results suggesting malignancy) were among the most important predictors for histology and aggressiveness. For complications, "ISMALE" (sex) emerged as a significant predictor.

Limitations and Future Work

  • Single-Center Retrospective Study: The authors acknowledge that the population is representative of a single region and that the retrospective nature may have led to missing data not recorded at surgery time.
  • Prospective Data Collection: They suggest that a prospective dataset collection would significantly improve the strength and generalizability of their findings.
  • Comparison with Clinical Assessment: An interesting future direction proposed is to compare the predictive model's sensitivity against the current clinical assessment derived from combining available data.
  • Multicentric and Prospective Analyses: The authors emphasize the need for future studies that are prospective, multicentric, and integrate clinical, laboratory, and ultrasound data to fully understand the clinical implications of ML in thyroid nodule management.

Summary updated

Abstract

Thyroid nodules are very common, 5-15% of which are malignant. Despite the low mortality rate of well-differentiated thyroid cancer, some variants may behave aggressively, making nodule differentiation mandatory. Ultrasound and fine-needle aspiration biopsy are simple, safe, cost-effective and accurate diagnostic tools, but have some potential limits. Recently, machine learning (ML) approaches have been successfully applied to healthcare datasets to predict the outcomes of surgical procedures. The aim of this work is the application of ML to predict tumor histology (HIS), aggressiveness and post-surgical complications in thyroid patients. This retrospective study was conducted at the ENT Division of Eastern Piedmont University, Novara (Italy), and reported data about 1218 patients who underwent surgery between January 2006 and December 2018. For each patient, general information, HIS and outcomes are reported. For each prediction task, we trained ML models on pre-surgery features alone as well as on both pre- and post-surgery data. The ML pipeline included data cleaning, oversampling to deal with unbalanced datasets and exploration of hyper-parameter space for random forest models, testing their stability and ranking feature importance. The main results are (i) the construction of a rich, hand-curated, open dataset including pre- and post-surgery features (ii) the development of accurate yet explainable ML models. Results highlight pre-screening as the most important feature to predict HIS and aggressiveness, and that, in our population, having an out-of-range (Low) fT3 dosage at pre-operative examination is strongly associated with a higher aggressiveness of the disease. Our work shows how ML models can find patterns in thyroid patient data and could support clinicians to refine diagnostic tools and improve their accuracy.
BibTeX
@article{dellera2023machine,
  author = {Valeria Dell’Era and Alan Perotti and Michele Starnini and Massimo Campagnoli and Maria Silvia Rosa and Irene Saino and Paolo Aluffi Valletti and Massimiliano Garzaro},
  title = {{Machine Learning Model as a Useful Tool for Prediction of Thyroid Nodules Histology, Aggressiveness and Treatment-Related Complications}},
  journal = {Journal of Personalized Medicine},
  year = {2023},
  doi = {10.3390/jpm13111615},
}
Figure