Comparison of Second-Order Polynomial Model Selection Methods: An Experimental Survey

AAAI 1999

TL;DR

This paper experimentally surveys various model selection methods for second-order polynomial models, comparing their ability to find parsimonious models that generalize well to new data. It finds that methods like MML, MDL, CAICF, SRM, and SC are robust candidates for automated model selection, while others like AIC and BIC tend to overfit.

What Problem Does This Paper Address?

When building predictive models, especially with complex data, a crucial challenge is selecting the "best" model. This involves finding a balance between how well the model fits the data it was trained on (goodness of fit) and how simple it is (complexity). Overly complex models can "memorize" the training data, leading to poor performance on new, unseen data (overfitting). Conversely, overly simple models might miss important patterns. This paper investigates various established and novel "model selection criteria" that aim to automate this balancing act, specifically for second-order polynomial models, which can capture non-linear relationships. The authors want to know which of these criteria are most reliable in practice, particularly for forecasting tasks like hurricane intensity change.

Methods

The study employs an experimental approach to evaluate several model selection methods. For each method, they developed a common gradient descent search strategy to find the optimal model parameters. The core of the work is a comparative analysis of these methods on both artificial and real-world datasets. The paper introduces a new method based on the Minimum Message Length (MML) principle and compares it against existing criteria.

Datasets

The paper uses two types of datasets:

  • Artificial Data: Generated from known "true" models to test the methods' ability to recover the underlying structure.
  • Real-World Data: Atmospheric data for hurricane intensity change forecasting. This dataset was generously provided by Chris Landsea of the Hurricane Research Centre NOAA, Miami, Florida.

Key Insights and Ideas

  • The Overfitting Trap: Many common model selection criteria (AIC, BIC, Cp, F-test, adjR2) tend to overfit the training data. This suggests their built-in penalties for model complexity aren't strong enough to counteract the gains from fitting the training data too closely.
  • Robust Candidates Emerge: Methods like MML, MDL, CAICF, SRM, and SC show promise as robust choices for automated model selection, reliably converging to a true or reasonably parsimonious model even with noisy data.
  • Test Data Isn't Enough for Some: PRESS and GCV criteria overfit the test data. This implies that relying solely on test set performance might not guarantee parsimonious models and highlights the need for a separate validation dataset.
  • Homogeneity is Key: The success of model selection methods hinges on having homogenous training and test datasets. This allows the patterns learned from the training data to be reliably extrapolated to new data.
  • Search Bias is Real: When dealing with large search spaces, non-exhaustive search strategies can introduce selection bias, influencing which local minimum a model selection method converges to.

Key Results

The paper demonstrates that for hurricane intensity change forecasting, MML, MDL, CAICF, SRM, and SC produced models with superior predictive performance compared to standard benchmark models in use. The experimental results highlight the practical implications of the theoretical insights, showing which methods are more reliable for real-world prediction tasks.

Limitations and Future Work

The authors acknowledge that the unavoidable practice of using non-exhaustive search strategies on large search spaces can introduce selection bias. They also note that methods like PRESS and GCV overfit test data, suggesting the need for a third, independent dataset for validation. Future work could explore more sophisticated search strategies to mitigate bias and further investigate the performance of robust methods across a wider range of complex forecasting problems.

Summary updated

BibTeX
@inproceedings{1999comparison,
  title = {{Comparison of Second-Order Polynomial Model Selection Methods: An Experimental Survey}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1999},
}
Figure