Population Gradients improve performance across data-sets and architectures in object classification

Yurika Sakai, Andrey Kormilitzin, Qiang Liu, Alejo J. Nevado-HolgadoarXiv 2020 · OpenReview

Introduction

Training deep neural networks (DNNs) typically involves minimizing an error function using empirical gradients. However, these gradients are calculated from finite datasets and specific weight configurations, making them susceptible to noise and local minima. While second-order optimization methods (e.g., using the Hessian matrix) offer a more global view of the error landscape, they are computationally prohibitive for modern architectures. This paper introduces Population Gradients (PG), a novel training method that approximates a "non-local" gradient by sampling from a population of noisy models, effectively smoothing the loss landscape and improving generalization.

The Population Gradients (PG) Method

The PG method replaces the standard single-point gradient update with a composite gradient derived from a neighborhood of the current weight configuration. The process follows four primary steps:

  1. Perturbation: At each training step, the central model's weights ($w_{original}$) are jittered to create a population of $k$ models. Noise is applied multiplicatively: $w_{i} = w_{original} \cdot (1 + \epsilon)$, where $\epsilon \sim \mathcal{N}(0, r)$ and $r$ is the "population range."
  2. Gradient Collection: Gradients are computed for each of the $k$ perturbed models.
  3. Combination: These gradients are averaged to produce a higher-quality, "smoothed" composite gradient.
  4. Update: The central model is updated using this averaged gradient.

By sampling the local neighborhood, PG acts as a stochastic approximation of the theoretical true gradient, ensuring that the update direction is robust to small-scale fluctuations in the loss surface.

Experimental Evaluation

The authors evaluated PG across five diverse object classification benchmarks: CIFAR-10, CIFAR-100, Fashion-MNIST, STL-10, and SVHN. To ensure architectural generalizability, the method was tested on five distinct models: DenseNet 161, MobileNet v2, ResNet 18, ResNet 50, and ShuffleNet v2.

Results and Analysis

Broad Performance Gains

PG consistently improved performance across all tested combinations of datasets and architectures. The method provided an average maximum F1 score increase of 5.3%. As shown in Figure 5, the performance gains are robust across different population sizes ($k$) and ranges ($r$), with optimal results typically found at $k=5$ and $r=0.2$.

F1 change across various architectures and datasets as a function of population size (k) and range (r).

Comparison with Standard Techniques

A key finding is that PG often outperforms or complements established regularization and optimization techniques. In head-to-head comparisons on CIFAR-10 and SVHN, PG ranked first among 11 enhancement methods, including AdamW, AMS-Grad, and standard L1/L2 regularization. Furthermore, PG exhibits additive benefits when combined with methods like weight decay or Dropout, suggesting it targets a unique aspect of model optimization.

Comparison of PG against standard regularization and optimization methods on ResNet 18 (CIFAR-10).

Robustness and Long-term Training

The authors also observed that PG is compatible with "double descent" phenomena, continuing to provide performance improvements even in extreme training regimes of up to 1000 epochs. This suggests that the smoothing effect of PG remains beneficial throughout the entire optimization process.

Conclusion

Population Gradients offer a simple yet powerful mechanism for improving DNN performance by smoothing the error landscape. While the method increases computational cost linearly with population size, the resulting gains in F1 score and robustness are significant. Future research will explore the application of PG in other domains such as Natural Language Processing and Reinforcement Learning.

Summary updated

Abstract

The most successful methods such as ReLU transfer functions, batch normalization, Xavier initialization, dropout, learning rate decay, or dynamic optimizers, have become standards in the field due, particularly, to their ability to increase the performance of Neural Networks (NNs) significantly and in almost all situations. Here we present a new method to calculate the gradients while training NNs, and show that it significantly improves final performance across architectures, data-sets, hyper-parameter values, training length, and model sizes, including when it is being combined with other common performance-improving methods (such as the ones mentioned above). Besides being effective in the wide array situations that we have tested, the increase in performance (e.g. F1) it provides is as high or higher than this one of all the other widespread performance-improving methods that we have compared against. We call our method Population Gradients (PG), and it consists on using a population of NNs to calculate a non-local estimation of the gradient, which is closer to the theoretical exact gradient (i.e. this one obtainable only with an infinitely big data-set) of the error function than the empirical gradient (i.e. this one obtained with the real finite data-set).
BibTeX
@misc{sakai2020population,
  author = {Yurika Sakai and Andrey Kormilitzin and Qiang Liu and Alejo J. Nevado-Holgado},
  title = {{Population Gradients improve performance across data-sets and architectures in object classification}},
  year = {2020},
  eprint = {2010.12260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.12260},
}
Figure