Evaluating Variable Selection Techniques for Multivariate Linear Regression

Nahyeon Ryu, Hyung Seok Kim, Pilsung KangJournal of Korean Institute of Industrial Engineers 2016 · DOI

Abstract

The purpose of variable selection techniques is to select a subset of relevant variables for a particular learning algorithm in order to improve the accuracy of prediction model and improve the efficiency of the model. We conduct an empirical analysis to evaluate and compare seven well-known variable selection techniques for multiple linear regression model, which is one of the most commonly used regression model in practice. The variable selection techniques we apply are forward selection, backward elimination, stepwise selection, genetic algorithm (GA), ridge regression, lasso (Least Absolute Shrinkage and Selection Operator) and elastic net. Based on the experiment with 49 regression data sets, it is found that GA resulted in the lowest error rates while lasso most significantly reduces the number of variables. In terms of computational efficiency, forward/backward elimination and lasso requires less time than the other techniques.

Introduction: The Curse of Dimensionality

In multivariate linear regression (MLR), practitioners often face the "curse of dimensionality." Including too many explanatory variables can lead to overfitting, where the model captures noise rather than the underlying signal, and multicollinearity, where variables are so highly correlated that they destabilize the model's coefficients.

The bias-variance tradeoff as a function of model dimensionality.

Variable selection aims to find the "Goldilocks" subset of predictors: enough to explain the data accurately, but few enough to keep the model efficient and interpretable. While many techniques exist—ranging from simple greedy searches to complex heuristics and penalized regression—there is little consensus on which performs best across different real-world scenarios. This study provides a rigorous ranking based on accuracy, sparsity, and speed.

Methods

The authors compare seven distinct approaches:

  1. Forward Selection: A greedy approach that starts with an empty model and adds the most significant variable one by one.
  2. Backward Elimination: Starts with all variables and removes the least significant ones.
  3. Stepwise Selection: A bidirectional approach that can both add and remove variables at each step.
  4. Genetic Algorithm (GA): A heuristic search inspired by natural selection that evolves "chromosomes" (subsets of variables) to minimize the Bayesian Information Criterion (BIC).
  5. Ridge Regression ($L_2$ penalty): Shrinks coefficients toward zero to handle multicollinearity.
  6. Lasso ($L_1$ penalty): Shrinks some coefficients exactly to zero, performing automatic variable selection.
  7. Elastic Net: A hybrid of Lasso and Ridge ($L_1 + L_2$ penalties).

The evaluation used the Bayesian Information Criterion (BIC) as the internal selection criterion to balance model fit against complexity.

Datasets

The study utilized 49 regression datasets sourced from the UCI Machine Learning Repository, the KEEL Dataset Repository, and the R project. These datasets vary significantly in scale, ranging from 7 to 95 variables and 32 to over 515,000 observations, ensuring robust results across diverse data types.

Key Insights and Ideas

  • The Global Search Advantage: GA consistently achieved the lowest error rates. Unlike greedy methods (Forward/Backward) that can get stuck in local optima, GA's stochastic nature allows it to explore a wider search space.
Comparison of local vs. global optima in search spaces.
  • The Sparsity Paradox: Lasso was the most aggressive at reducing the number of variables, but this often led to the highest error rates. This suggests that Lasso might "over-prune" relevant information in its quest for a sparse model.
  • Ridge's Hidden Strength: Despite not being a "selection" method by design, Ridge ranked 2nd in error rate improvement, highlighting the power of regularization for generalization.
  • Efficiency vs. Quality: There is a massive disparity in computational cost. GA could take up to 10 hours on large datasets, whereas Forward Selection or Lasso typically finished in seconds.

Key Results

The techniques were ranked across three criteria (1 is best, 7 is worst):

Technique Error Rate Improvement Variable Reduction Rate Computational Efficiency
GA 1 6 7
Ridge 2 7 5
Stepwise 3 2 6
Backward 4 3 2
Forward 5 4 1
Elastic Net 6 5 4
Lasso 7 1 3

Limitations and Future Work

The study is limited to linear models and fixed hyperparameters (e.g., GA generations, Elastic Net alpha). Future work could explore these selection techniques within non-linear frameworks like Support Vector Regression or Neural Networks to see if these rankings hold.

Summary updated

Abstract

The purpose of variable selection techniques is to select a subset of relevant variables for a particular learning algorithm in order to improve the accuracy of prediction model and improve the efficiency of the model. We conduct an empirical analysis to evaluate and compare seven well-known variable selection techniques for multiple linear regression model, which is one of the most commonly used regression model in practice. The variable selection techniques we apply are forward selection, backward elimination, stepwise selection, genetic algorithm (GA), ridge regression, lasso (Least Absolute Shrinkage and Selection Operator) and elastic net. Based on the experiment with 49 regression data sets, it is found that GA resulted in the lowest error rates while lasso most significantly reduces the number of variables. In terms of computational efficiency, forward/backward elimination and lasso requires less time than the other techniques.
BibTeX
@article{ryu2016evaluating,
  author = {Nahyeon Ryu and Hyung Seok Kim and Pilsung Kang},
  title = {{Evaluating Variable Selection Techniques for Multivariate Linear Regression}},
  journal = {Journal of Korean Institute of Industrial Engineers},
  year = {2016},
  doi = {10.7232/jkiie.2016.42.5.314},
}
Figure