Automatic Variable Selection Algorithms in Prognostic Factor Research in Neck Pain

Bernard X. W. Liew, Francisco M. Kovacs, David Rügamer, Ana RoyuelaJournal of Clinical Medicine 2023 · DOI

TL;DR

This study compares various machine learning and statistical algorithms for selecting prognostic factors in neck pain recovery. It finds that while most algorithms offer similar predictive performance, they differ significantly in the number of variables they select, with traditional stepwise methods sometimes missing important predictors. The multivariate adaptive regression spline (MuARS) algorithm emerges as a strong contender, balancing model simplicity with predictive power.

What Problem Does This Paper Address?

Identifying reliable predictors for neck pain (NP) recovery is crucial for effective treatment, but the multifactorial nature of NP makes this challenging. Prognostic factor research often relies on statistical models, with stepwise regression being a common, yet potentially flawed, method for variable selection. Stepwise methods can inflate the significance of variables, leading to models that are less interpretable and may have reduced predictive performance on new data. Furthermore, there's a lack of comparative studies directly assessing how different machine learning (ML) algorithms, particularly those with built-in variable selection capabilities, stack up against traditional methods in this specific clinical context. This paper aims to fill that gap by comparing eight distinct algorithms on their variable selection strategies and their impact on predictive performance for NP, arm pain (AP), and disability outcomes.

Methods

The study employed a longitudinal observational design with data collected at baseline and 3-month follow-up from 3001 participants with NP. They compared eight variable selection algorithms:

  • Traditional Stepwise Regression:
    • stepP: Based on unadjusted p-values.
  • stepPAdj: Based on adjusted p-values.
  • stepAIC: Based on Akaike Information Criterion.
  • Other Statistical/ML Algorithms:
    • BestSubset: Exhaustively searches for the best model of a given size.
  • LASSO: A regularized regression method that shrinks coefficients and can set some to zero.
  • MCP (Minimax Concave Penalty): Another regularized regression method.
  • mboost (model-based boosting): An ensemble method that iteratively builds models.
  • MuARS (Multivariate Adaptive Regression Splines): A flexible non-parametric regression method.

The data was split into training (80%) and testing (20%) sets, with missing data handled using multiple imputation. Continuous predictors were scaled, and categorical predictors were one-hot encoded. Model performance was evaluated using Area Under the Curve (AUC), accuracy, precision, sensitivity, and specificity. The researchers also focused on the number of selected predictors, their coefficient magnitudes, and signs.

Datasets

The study utilized a dataset of 3001 participants with non-specific neck pain, collected from 47 healthcare centers in Spain. The data included baseline characteristics, diagnostic procedures, treatments, and outcome measures (neck pain improvement, arm pain improvement, and disability improvement) at 3-month follow-up. No new benchmark datasets were introduced; the focus was on applying and comparing algorithms to an existing clinical dataset.

Key Insights and Ideas

  • Stepwise Regression's Shortcomings: Traditional stepwise regression, especially when using unadjusted p-values, didn't necessarily produce the sparsest models. More critically, stepPAdj (stepwise with adjusted p-values) resulted in the sparsest models but, for the disability outcome, showed poorer predictive performance, suggesting that over-simplification can be detrimental.
  • Algorithm Diversity in Variable Selection: While most algorithms yielded similar predictive performance, they varied considerably in the number of variables they selected. This highlights that different methods can arrive at very different "optimal" models for the same data.
  • The "Always Selected" Predictors: Certain variables, like "neuroreflexotherapy intervention" for NP and AP outcomes, and "imaging findings: spinal stenosis" for disability, were consistently selected by nearly all algorithms. This suggests these are robust prognostic indicators.
  • MuARS as a Promising Candidate: The MuARS algorithm consistently selected a relatively small number of predictors (second fewest after stepPAdj) while maintaining high predictive performance across all outcomes. This suggests it strikes a good balance between model interpretability and predictive accuracy.
  • Confidence Through Ensemble: The paper advocates for using multiple algorithms to select variables. This approach can increase confidence in the identified prognostic factors, as variables consistently selected across diverse methods are likely to be more robust.

Key Results

  • Sparsest Models: stepPAdj produced the sparsest models (4-8 predictors). MuARS was the next sparsest (9-14 predictors).
  • Densest Models: LASSO and BestSubset tended to select the most predictors (up to 28 for AP/disability and NP, respectively).
  • Consistent Predictors: Three predictors were selected by all eight algorithms for NP and AP outcomes, and six for disability.
  • Predictive Performance: The differences in predictive performance (AUC, accuracy, etc.) between the best and worst performing algorithms were generally small for NP and AP outcomes (differences of 0.01-0.06). However, for disability, the differences were more pronounced, particularly for sensitivity (0.23).
  • Coefficient Magnitudes: LASSO and mboost coefficients were, on average, smaller than their refitted counterparts, indicating the shrinkage effect of these methods.

Limitations and Future Work

  • Limited Algorithm Scope: The study did not investigate all possible ML algorithms with embedded variable selection, notably excluding classification or regression trees.
  • Post-Selection Inference: The study did not provide statistical inference (e.g., confidence intervals) for the selected variables, acknowledging that valid post-selection inference is challenging due to the data-driven nature of variable selection.
  • Generalizability: While the study used a large dataset, future work could explore the generalizability of these findings to different populations or pain conditions.
  • Ensemble Methods: The authors suggest that future research should focus on developing ensemble methods that combine multiple variable selection strategies to enhance stability and robustness. Investigating methods to quantify selection stability across algorithms is also recommended.

Summary updated

Abstract

This study aims to compare the variable selection strategies of different machine learning (ML) and statistical algorithms in the prognosis of neck pain (NP) recovery. A total of 3001 participants with NP were included. Three dichotomous outcomes of an improvement in NP, arm pain (AP), and disability at 3 months follow-up were used. Twenty-five variables (twenty-eight parameters) were included as predictors. There were more parameters than variables, as some categorical variables had >2 levels. Eight modelling techniques were compared: stepwise regression based on unadjusted <i>p</i> values (stepP), on adjusted <i>p</i> values (stepPAdj), on Akaike information criterion (stepAIC), best subset regression (BestSubset) least absolute shrinkage and selection operator [LASSO], Minimax concave penalty (MCP), model-based boosting (mboost), and multivariate adaptive regression splines (MuARS). The algorithm that selected the fewest predictors was stepPAdj (number of predictors, <i>p</i> = 4 to 8). MuARS was the algorithm with the second fewest predictors selected (<i>p</i> = 9 to 14). The predictor selected by all algorithms with the largest coefficient magnitude was "having undergone a neuroreflexotherapy intervention" for NP (β = from 1.987 to 2.296) and AP (β = from 2.639 to 3.554), and "Imaging findings: spinal stenosis" (β = from -1.331 to -1.763) for disability. Stepwise regression based on adjusted <i>p</i>-values resulted in the sparsest models, which enhanced clinical interpretability. MuARS appears to provide the optimal balance between model sparsity whilst retaining high predictive performance across outcomes. Different algorithms produced similar performances but resulted in a different number of variables selected. Rather than relying on any single algorithm, confidence in the variable selection may be increased by using multiple algorithms.
BibTeX
@article{liew2023automatic,
  author = {Bernard X. W. Liew and Francisco M. Kovacs and David Rügamer and Ana Royuela},
  title = {{Automatic Variable Selection Algorithms in Prognostic Factor Research in Neck Pain}},
  journal = {Journal of Clinical Medicine},
  year = {2023},
  doi = {10.3390/jcm12196232},
}
Figure