Hyperparameters Evidence and Generalisation for an Unrealisable Rule

Glenn Marion, David SaadNeurIPS 1994

Abstract

This paper investigates whether the Bayesian "evidence procedure"—a popular method for tuning hyperparameters—actually works when your model is fundamentally too simple to represent the truth (the "unrealisable" case). Using a statistical mechanics framework, the authors show that while the procedure becomes sub-optimal when learning non-linear rules with a linear model, it remains a surprisingly robust and practical heuristic.

Introduction: The Problem of Unrealisable Rules

In Bayesian machine learning, we often use the evidence procedure (also known as Type II Maximum Likelihood) to set hyperparameters like weight decay or noise variance. This approach assumes that the "true" rule generating the data exists somewhere within our model’s hypothesis space.

But what happens in the real world, where our models are almost always approximations? This is the "unrealisable" scenario. If a linear perceptron tries to learn a non-linear rule, the "true" rule is outside its reach. The authors ask: Does the evidence procedure still guide us to the best possible generalization performance, or does the model-teacher mismatch lead us astray?

Methodology: A Statistical Mechanics Approach

The authors employ a statistical mechanical formalism to analyze the behavior of a linear perceptron (the student) learning from a more complex "n-teacher."

  • The Student: A simple linear perceptron.
  • The Teacher: A non-linear entity modeled as a mixture of linear components (an "n-teacher"). By adjusting the parameters of this teacher, the authors can interpolate between a perfectly learnable linear rule and a complex, non-linear, unrealisable rule.
  • The Math: They calculate the "evidence" (the marginal likelihood), generalization error, and a "consistency measure" in the thermodynamic limit (where the number of inputs $N$ and examples $p$ both go to infinity, but their ratio $\alpha = p/N$ remains constant). This allows them to derive analytical expressions for how the model behaves on average.

The study uses synthetic datasets generated specifically for this theoretical analysis. Inputs are sampled from a Gaussian distribution, and outputs are generated by the "n-teacher" (specifically a 2-teacher model for the results shown), which maps inputs to outputs according to a probabilistic distribution that includes both Gaussian noise and structural non-linearity. This synthetic approach allows the authors to precisely control the "level of unlearnability" and the noise level, which would be impossible with real-world data.

Key Insights: Effective Noise and the Generalization Hump

The most striking insight is that a mismatch between the student and teacher acts almost exactly like added noise. Even if the teacher is deterministic, the student perceives the unmodelable non-linearity as random fluctuations.

Generalisation error and consistency measure

When the number of examples ($\alpha$) is small, the generalization error actually increases before it starts to decrease, creating a "generalization hump." This happens because the student initially "learns" the effective noise (the non-linearity it can't understand) before it gets enough data to average it out and find the best linear approximation.

Furthermore, the evidence procedure breaks down. In the linear (realisable) case, the hyperparameters that maximize evidence also perfectly minimize generalization error. In the non-linear (unrealisable) case, these two goals diverge. The evidence procedure picks hyperparameters that are "wrong" for optimal performance. Despite the theoretical breakdown, the actual performance penalty (the "degradation") is often quite small—frequently only a few percent.

Results: Evidence Breakdown in Non-Linear Scenarios

The results clearly contrast the realisable and unrealisable learning scenarios.

Optimal hyperparameters in linear vs non-linear cases
  • Linear Case: The evidence procedure is optimal. The curves for maximum evidence, minimum generalization error, and optimal consistency all intersect at a single point.
  • Non-Linear Case: The evidence procedure fails to find the absolute minimum generalization error. The hyperparameter assignments become dependent on the number of training examples ($\alpha$), which shouldn't happen in a perfectly specified Bayesian model.

To quantify this failure, the authors define $\kappa$ as the percentage degradation in performance. For relatively large deviations from linearity, the evidence procedure remains "close to optimal," suggesting it is a "reasonable, if not optimal, method" for practical use.

Limitations and Future Directions

The study is limited to a linear student. While this allows for exact mathematical solutions, modern neural networks are non-linear and may exhibit more complex failure modes under the evidence procedure. Additionally, the analysis relies heavily on Gaussian noise and input distributions.

Future work could investigate whether this breakdown is a "generic feature" of all model-teacher mismatches. The authors also note that while the evidence procedure is "good enough," the real solution is to improve the student's hypothesis space (i.e., use a more complex model) so the rule becomes realisable.

Summary updated

BibTeX
@inproceedings{marion1994hyperparameters,
  author = {Glenn Marion and David Saad},
  title = {{Hyperparameters Evidence and Generalisation for an Unrealisable Rule}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1994},
}
Figure