VECA: A Method for Detecting Overfitting in Neural Networks (Student Abstract)

Liangzhu Ge, Yuexian Hou, Yaju Jiang, Shuai Yao, Chao YangAAAI 2020 · DOI

Introduction

In the field of deep learning, overfitting is a pervasive issue that practitioners typically mitigate using a validation set. By monitoring the error on data not seen during training, one can determine when a model begins to memorize noise rather than learning generalizable patterns. However, in "small data" regimes—such as medical imaging or specialized industrial tasks—sacrificing a significant portion of the dataset for validation is a luxury that researchers often cannot afford.

In this student abstract from AAAI-20, Liangzhu Ge and colleagues propose VECA (Variance of Eigenvalues of Covariance matrix of Activation matrix), an internal metric that predicts overfitting using only training data. By analyzing the redundancy of features in the network's last hidden layer, VECA signals the onset of overfitting without requiring a separate validation set.

Methodology

The authors focus on the network's last hidden layer, which acts as the final feature representation before the output layer. The core hypothesis is that overfitting is linked to a structural change in how the network perceives data, specifically an increase in feature redundancy.

The VECA metric is calculated through the following steps:

  1. Activation Matrix: Record the activations of the $n$ units in the last hidden layer for a sample of training data.
  2. Covariance Matrix ($C_Z$): Compute the covariance matrix of these activations to capture both individual unit variance and inter-unit correlations.
  3. Eigenvalue Decomposition: Determine the eigenvalues ($\lambda$) of the covariance matrix. These eigenvalues represent the variance captured by each principal direction in the feature space.
  4. VECA Calculation: VECA is defined as the variance of these eigenvalues.

The intuition is that a well-generalizing network utilizes its feature space uniformly (low VECA). As it begins to overfit, it often becomes hyper-reliant on specific directions, causing a few eigenvalues to spike while others drop toward zero, thereby increasing the VECA score.

Experimental Results

The researchers evaluated VECA using the Fashion-MNIST dataset across both Fully-Connected Networks (FCNs) and Convolutional Neural Networks (CNNs). By training 50 models with varying hyperparameters, they identified a consistent relationship between internal activation dynamics and generalization error.

Figure 1: Comparison of Test Loss and VECA over 100 training epochs. The rise in VECA closely tracks the increase in test loss.

As illustrated in Figure 1, there is a strong positive correlation between test loss and the VECA score. The moment VECA begins to rise sharply almost perfectly coincides with the point at which test loss begins to climb, signaling the start of overfitting.

Figure 2: Consistent tracking of overfitting by VECA over 160 training epochs.

This predictive timing remains robust across different training durations (see Figure 2). The authors conclude that for any two points in training with similar training loss, the one with the larger VECA value is almost certainly the one with inferior generalization ability.

Discussion and Future Directions

VECA provides a mathematically rigorous way to quantify the "Single Direction Hypothesis," which suggests that overfit networks rely on a limited number of activation directions. As an internal "canary in the coal mine," it allows for real-time monitoring of generalization health without touching a validation set.

Future work could explore the scalability of VECA to larger architectures like Transformers or more complex datasets like ImageNet. Additionally, the authors suggest that VECA could be used as a regularizer: by adding VECA as a penalty term in the loss function, one could force the network to maintain a diverse, non-redundant feature space, effectively embedding anti-overfitting measures directly into the optimization process.

Summary updated

BibTeX
@inproceedings{ge2020veca,
  author = {Liangzhu Ge and Yuexian Hou and Yaju Jiang and Shuai Yao and Chao Yang},
  title = {{VECA: A Method for Detecting Overfitting in Neural Networks (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7167},
}
Figure