A Surprising Linear Relationship Predicts Test Performance in Deep Networks

Qianli Liao, Brando Miranda, Andrzej Banburski, Jack Hidary, Tomaso A. PoggioarXiv 2018 · arXiv

Introduction: The Generalization Puzzle

In deep learning, training loss is often a deceptive metric. Overparametrized networks can achieve near-zero training loss even on randomly labeled data, yet they fail to generalize to a test set. This phenomenon leads to a "generalization puzzle": given two networks with identical training loss, why do they exhibit drastically different test results? If training loss cannot reliably predict test performance, how can we choose between different architectures or training regimes?

Training loss remains near zero even as test error increases with data corruption.

Methods: Normalizing the Weights

The authors propose that the "noise" in training loss stems from the scale of the weights. In ReLU-based networks, which exhibit "positive homogeneity," scaling the weights of a layer by a factor $\rho$ scales the output. Stochastic Gradient Descent (SGD) can minimize cross-entropy loss simply by increasing weight magnitudes (increasing the network's "confidence") without necessarily improving decision boundaries.

To "fix" this, the researchers applied layerwise weight normalization. By normalizing each layer’s weight matrix (typically using the Frobenius norm) to a constant value, they factored out the scale. This allowed them to calculate a Normalized Cross-Entropy loss, which effectively separates classification performance from the weight-scaling effects that SGD minimizes but which do not contribute to generalization.

Inducing and Measuring Generalization Gaps

To test their hypothesis, the authors intentionally created models with identical training losses but divergent test errors using two techniques:

  1. Random Pretraining: Training on "corrupted" data (shuffled labels) before switching to clean data.
  2. Initialization Variance: Initializing weights with different standard deviations.
Varying weight initialization creates models with identical training loss but divergent test performance.

As shown in the figures above, while the training loss (blue circles) remains consistently near zero across different corruption levels or initialization variances, the test error (orange triangles) increases significantly. This confirms that raw training loss is a poor proxy for generalization.

Key Insights: The Hidden Linearity

The most striking result of this study is that once weights are normalized, the relationship between training loss and test loss becomes nearly perfectly linear. In experiments on the CIFAR-10 dataset, the normalized training vs. test loss showed an $R^2$ value of 0.9998.

This linear relationship held true regardless of the training regime. Even when a network was trained on 100% random labels—achieving 0% training error but chance-level test error—the normalized loss point fell exactly on the same linear path as models trained on clean data. This suggests that the underlying capacity of the network and its error are tied together in a predictable way once the "scale" of the weights is accounted for.

Conclusion and Practical Implications

The findings suggest that theoretical generalization bounds, often criticized for being too "loose" for practical use, are actually very tight when viewed through the lens of normalization. For practitioners, the takeaway is clear: monitor normalized cross-entropy loss during training. This metric provides a much more reliable signal for deciding when to stop training or which model version to deploy, as it directly reflects the model's ability to generalize beyond the training set.

While the study focused on standard benchmarks like MNIST and CIFAR, the theoretical basis of homogeneity suggests these findings likely extend to many modern architectures. Future work may explore how these relationships hold in even deeper structures like Transformers.

Summary updated

Abstract

Given two networks with the same training loss on a dataset, when would they have drastically different test losses and errors? Better understanding of this question of generalization may improve practical applications of deep networks. In this paper we show that with cross-entropy loss it is surprisingly simple to induce significantly different generalization performances for two networks that have the same architecture, the same meta parameters and the same training error: one can either pretrain the networks with different levels of "corrupted" data or simply initialize the networks with weights of different Gaussian standard deviations. A corollary of recent theoretical results on overfitting shows that these effects are due to an intrinsic problem of measuring test performance with a cross-entropy/exponential-type loss, which can be decomposed into two components both minimized by SGD -- one of which is not related to expected classification performance. However, if we factor out this component of the loss, a linear relationship emerges between training and test losses. Under this transformation, classical generalization bounds are surprisingly tight: the empirical/training loss is very close to the expected/test loss. Furthermore, the empirical relation between classification error and normalized cross-entropy loss seem to be approximately monotonic
BibTeX
@misc{liao2018surprising,
  author = {Qianli Liao and Brando Miranda and Andrzej Banburski and Jack Hidary and Tomaso A. Poggio},
  title = {{A Surprising Linear Relationship Predicts Test Performance in Deep Networks}},
  year = {2018},
  eprint = {1807.09659},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.09659},
}
Figure