Models and Selection Criteria for Regression and Classification

David Heckerman, Christopher Meek

TL;DR

This paper explores Bayesian approaches to regression and classification, focusing on how to select the best models. It introduces Bayesian Regression/Classification (BRC) models and critically examines the practice of transforming general Bayesian networks into these simpler forms, arguing it can lead to overlooking crucial prior knowledge. The authors also compare two prequential criteria for model selection, identifying conditions under which they agree.

What Problem Does This Paper Address?

When building predictive models, especially in machine learning, we often want to predict an outcome variable (Y) given a set of input variables (X). This is the core of regression (continuous Y) and classification (discrete Y). While Bayesian networks are powerful tools for modeling the joint probability distribution of all variables, directly using them for regression/classification can be computationally intensive. A common simplification is to use "conditional models" that focus solely on $p(y|x)$. This paper investigates a specific class of Bayesian models designed for this task, called Bayesian Regression/Classification (BRC) models, and critically analyzes the common practice of deriving these models by "embedding" a conditional likelihood from a more complex joint model. The authors highlight that this embedding process might discard valuable information about the relationships between input variables and model parameters, potentially hindering learning. Furthermore, the paper addresses the challenge of selecting the "best" Bayesian model for this task from a set of candidates, comparing different scoring criteria.

Methods

The paper introduces Bayesian Regression/Classification (BRC) models, which are a special class of Bayesian models that can be factored into independent conditional ($y|x$) and input ($x$) models. This factorization simplifies analysis, as the conditional part can be studied in isolation.

A key methodological contribution is the critique of embedding BRC models. This involves taking a general Bayesian network model and extracting its conditional likelihood $p(y|x, \theta_m, m)$, then using it within a BRC framework. The authors argue this practice can be problematic because it might ignore "variational dependencies" – situations where the parameters of the conditional model ($\theta_{yx}$) are not independent of the parameters governing the input variables ($\theta_x$). This dependence means that observing input data (X) can actually inform our beliefs about the parameters of the conditional model, a nuance lost in the simple embedding approach.

The paper also delves into Bayesian model selection for regression/classification. It compares two prequential (predictive sequential) criteria: the Conditional Node Monitor (CNM) and the Class Sequential Criterion (CSC). These criteria evaluate models based on their ability to predict data sequentially. The authors provide mathematical derivations and analyze the conditions under which these two criteria yield the same results, finding they agree for BRC models.

Datasets

Not discussed in the paper. The paper focuses on theoretical aspects and model comparisons rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • BRC models offer a convenient simplification: By separating the conditional ($y|x$) and input ($x$) models, BRC models make it easier to focus on the predictive task of regression/classification.
  • Embedding BRC models can be problematic: Extracting a conditional likelihood from a complex joint model and treating it as a standalone BRC model can lead to ignoring important "variational dependencies." This means that observations of input variables might actually inform our beliefs about the parameters of the conditional model, a subtlety lost in the simplified BRC approach.
  • Prior knowledge can be lost: The practice of embedding BRC models might discard prior knowledge encoded in the relationships between input variables and model parameters within the original joint model.
  • Two prequential criteria for model selection exist: The CNM and CSC criteria offer different ways to evaluate models based on their sequential predictive performance.
  • BRC models make prequential criteria agree: A significant finding is that under the assumption that a model is a BRC model, the CNM and CSC criteria become equivalent, simplifying model selection in these cases.

Key Results

The paper provides theoretical comparisons of model selection criteria and analyzes the properties of BRC models. While no quantitative results on specific datasets are presented, the key theoretical results include:

  • Derivations showing how to transform a naive Bayes model into a linear softmax regression model.
  • Mathematical conditions under which the Conditional Node Monitor (CNM) and Class Sequential Criterion (CSC) for Bayesian model selection are equivalent. This equivalence holds when the model in question is a BRC model.

Limitations and Future Work

  • Theoretical objections to BERC models: The authors express theoretical concerns about using non-trivial Bayesian Embedded Regression/Classification (BERC) models, particularly when they are derived from more complex models. They argue that these models might ignore important variational dependencies.
  • Need for empirical investigation: Despite theoretical objections, the authors acknowledge that BERC models (like polynomial softmax regressions) might still lead to better predictions in practice due to their parsimony. They suggest that experiments are needed to investigate these possibilities.
  • Generalization of criteria: While the paper focuses on logarithmic utility functions for prequential criteria, it notes that other utility functions might be more appropriate for certain problems.
  • Computational complexity: The paper mentions that computing the CSC criterion can be computationally expensive (exponential in sample size) for large datasets, suggesting Monte-Carlo or asymptotic techniques might be necessary.

Summary updated

Abstract

When performing regression or classification, we are interested in the conditional probability distribution for an outcome or class variable Y given a set of explanatoryor input variables X. We consider Bayesian models for this task. In particular, we examine a special class of models, which we call Bayesian regression/classification (BRC) models, that can be factored into independent conditional (y|x) and input (x) models. These models are convenient, because the conditional model (the portion of the full model that we care about) can be analyzed by itself. We examine the practice of transforming arbitrary Bayesian models to BRC models, and argue that this practice is often inappropriate because it ignores prior knowledge that may be important for learning. In addition, we examine Bayesian methods for learning models from data. We discuss two criteria for Bayesian model selection that are appropriate for repression/classification: one described by Spiegelhalter et al. (1993), and another by Buntine (1993). We contrast these two criteria using the prequential framework of Dawid (1984), and give sufficient conditions under which the criteria agree.
BibTeX
@inproceedings{heckerman1997models,
  author = {David Heckerman and Christopher Meek},
  title = {{Models and Selection Criteria for Regression and Classification}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {1997},
}
Figure