Models and Selection Criteria for Regression and Classification

David Heckerman, Christopher Meek

TL;DR

This paper explores the theoretical nuances of Bayesian regression and classification, specifically warning against the common practice of ignoring the input distribution $p(\mathbf{x})$ when learning a conditional model $p(y|\mathbf{x})$. The authors demonstrate that "variational dependencies" between parameters in joint models can provide valuable information for learning, and they provide a formal comparison of two key Bayesian model selection criteria using the prequential framework.

What Problem Does This Paper Address?

In machine learning, we often care only about predicting an outcome $Y$ given inputs $\mathbf{X}$. Because of this, researchers frequently use "conditional models" (like logistic regression or softmax) that ignore how the inputs $\mathbf{X}$ are distributed.

Heckerman and Meek tackle two main questions:

  1. Is it safe to ignore the input distribution? Many researchers take a complex joint model (like a Naive Bayes network) and extract just the conditional part to simplify analysis. The authors ask if this "embedding" process loses critical information.
  2. How should we choose between models? When our goal is specifically classification or regression, standard "global" model selection criteria (which look at the whole joint distribution) might not be appropriate. The authors compare "local" criteria designed specifically for these tasks.

Methods

The authors define a specific class of models called Bayesian Regression/Classification (BRC) models. In a BRC model, the parameters for the input distribution $p(\mathbf{x})$ and the conditional distribution $p(y|\mathbf{x})$ are independent.

They contrast this with Bayesian Embedded Regression/Classification (BERC) models. A BERC model is created by taking a joint model (like the Naive Bayes model in Figure 1a) and extracting its conditional likelihood to create a standalone regression model (like the softmax model in Figure 1b).

Figure 1: ( a) A naive Bayes model for classification. (b) A linear soft max regression model that has the same conditional distribution for Y.

To evaluate model selection, they use Dawid’s prequential framework, which views model scores as a measure of how well a model sequentially predicts data. They specifically compare:

  • Conditional Node Monitor (CNM): A criterion that looks at how well the model predicts $y_i$ given $x_i$ and all previous pairs of $(x, y)$.
  • Class Sequential Criterion (CSC): A criterion that assumes all inputs $\mathbf{X}$ are seen first, then evaluates the sequential prediction of the labels $Y$.

Datasets

The paper is primarily theoretical and methodological. The authors mention applying their criteria to "small Bayesian networks and small data sets chosen arbitrarily" to confirm that the selection criteria differ in practice, but they do not use or introduce specific named benchmark datasets.

Key Insights and Ideas

  • Variational Dependence: This is the paper's most significant theoretical contribution. The authors argue that in many joint models, the parameters governing $\mathbf{X}$ and those governing $Y|\mathbf{X}$ are linked. If you observe $\mathbf{X}$, you actually learn something about the parameters of $Y|\mathbf{X}$. By using a BERC model (Figure 2), you sever this link, potentially ignoring prior knowledge that could speed up learning.
Figure 2: A BERC model obtained from a naive-Bayes
  • The Jacobian Rank Argument: Using differential geometry concepts, they show that for almost all Naive Bayes models, the mapping from joint parameters to input parameters has a full-rank Jacobian. This implies that the parameters are variationally dependent—knowing one constrains the other.
  • Criteria Convergence: While the CNM and CSC criteria generally yield different results, the authors prove they are mathematically identical if (and only if) the model is a BRC model.

Key Results

  • Theoretical Critique: The authors successfully argue that transforming a Bayesian network into a simple conditional model (like a softmax regression) is "often inappropriate" because it assumes parameter independence that doesn't exist in the original joint distribution.
  • Selection Criteria Comparison: They provide the first formal comparison between Spiegelhalter’s CNM and Buntine’s CSC. They show that while CNM is computationally efficient under certain assumptions (Dirichlet priors), CSC is significantly more complex to compute (exponential in sample size $N$), requiring Monte-Carlo or asymptotic approximations for large datasets.

Limitations and Future Work

The authors acknowledge a major caveat: while BERC models are theoretically "flawed" because they ignore variational dependencies, they are often much simpler (more parsimonious). In the real world, a simpler, slightly "wrong" model often generalizes better than a complex, "correct" one.

The authors suggest that future experimental work is needed to determine if the theoretical benefits of maintaining joint distributions outweigh the practical benefits of the simpler, parsimonious BERC models (like polynomial softmax regressions). They also note the need for better computational methods for the CSC criterion in large-scale applications.

Summary updated

Abstract

When performing regression or classification, we are interested in the conditional probability distribution for an outcome or class variable Y given a set of explanatoryor input variables X. We consider Bayesian models for this task. In particular, we examine a special class of models, which we call Bayesian regression/classification (BRC) models, that can be factored into independent conditional (y|x) and input (x) models. These models are convenient, because the conditional model (the portion of the full model that we care about) can be analyzed by itself. We examine the practice of transforming arbitrary Bayesian models to BRC models, and argue that this practice is often inappropriate because it ignores prior knowledge that may be important for learning. In addition, we examine Bayesian methods for learning models from data. We discuss two criteria for Bayesian model selection that are appropriate for repression/classification: one described by Spiegelhalter et al. (1993), and another by Buntine (1993). We contrast these two criteria using the prequential framework of Dawid (1984), and give sufficient conditions under which the criteria agree.
BibTeX
@inproceedings{heckerman1997models,
  author = {David Heckerman and Christopher Meek},
  title = {{Models and Selection Criteria for Regression and Classification}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {1997},
}
Figure