Abstract
Standard deep learning optimization via mini-batch Stochastic Gradient Descent (SGD) often suffers from inefficient "zig-zag" convergence due to its reliance on local data distributions. We present Epoch-evolving Gaussian Process Guided Learning (GPGL), a novel framework that incorporates global data distribution information into the training process. By utilizing a Gaussian Process (GP) to estimate "context labels" from a representative anchor set, GPGL provides a top-down guidance signal that regularizes the learning process. Experimental results on CIFAR and Tiny-ImageNet benchmarks demonstrate that GPGL achieves state-of-the-art accuracy while significantly accelerating convergence, often reaching peak performance 50 to 90 epochs faster than standard SGD-Momentum.
Introduction
Deep neural networks are typically trained using mini-batch SGD, where model parameters are updated based on small, randomly sampled subsets of the training data. While computationally efficient, this "bottom-up" approach ignores the global structure of the dataset during individual updates. This leads to two primary inefficiencies: the "zig-zag" effect, where optimization paths become noisy as the model overfits to local batch quirks, and slow overall convergence as the model takes many iterations to synthesize a global understanding of the data.
To address these issues, we propose GPGL, which introduces a "top-down" strategy. By maintaining a sense of the global data distribution through a nonparametric Gaussian Process model, we can provide each training sample with a "context label" that reflects its relationship to the broader dataset, smoothing the optimization landscape and guiding the model toward more robust representations.
Methodology
Context Label Estimation and Anchor Sets
The core of GPGL is the generation of context labels. We first construct an Anchor Set, a representative subset of the global data that serves as a proxy for the entire distribution. For any given training sample, a Gaussian Process regression model calculates a context label by evaluating the sample's proximity to these anchors in the current feature space. This process effectively propagates labels from the global anchor set to the local batch, providing a smoother supervision signal than standard one-hot labels.
Triangle Consistency Loss
To integrate this guidance, we employ a three-way "triangle" loss function:
- Standard Cross-Entropy ($L^1_{ce}$): Measures the discrepancy between the model's prediction and the ground truth label.
- KL-Divergence Loss ($L_{kl}$): Regularizes the model by forcing its predictions to align with the GP-generated context labels.
- Feature Consistency Loss ($L^2_{ce}$): Compares the context label directly to the ground truth, specifically updating the feature extractor to ensure the learned representation space remains consistent with actual class boundaries.
Epoch-Evolving Updates
Because the feature space evolves as the network learns, the GP model and the resulting context labels are updated at the end of every epoch. This creates a dynamic regularization effect where the guidance becomes increasingly precise as the model's internal representations improve.
Experimental Results
We evaluated GPGL across several standard benchmarks, including CIFAR-10, CIFAR-100, and Tiny-ImageNet, using various ResNet and ResNeXt architectures.
Performance and Convergence
GPGL consistently outperformed standard SGD-Momentum (SGD-M) in both final accuracy and training efficiency. On CIFAR-100, GPGL provided an accuracy boost of up to 1.47%. More importantly, the framework demonstrated remarkable convergence speed. As shown in the validation error plots, GPGL reaches stable, high-performance regimes much earlier in the training cycle than baseline methods.
On the challenging Tiny-ImageNet dataset with a ResNet18 backbone, GPGL achieved superior results in just 49 epochs compared to the 115 epochs required by SGD-M. This acceleration suggests that the global guidance provided by the context labels effectively mitigates the noisy updates inherent in standard mini-batch training.
Conclusion
GPGL introduces a powerful nonparametric guidance mechanism to deep learning optimization. By bridging the gap between local batch updates and global data distributions via Gaussian Process regression, the method reduces optimization noise and drastically cuts training time. While the computational cost of the GP scales with the anchor set size, class-aware sampling makes the approach feasible for modern benchmarks. Future work will explore scaling GPGL to larger datasets like ImageNet-1k using sparse GP techniques and extending the triangle consistency framework to non-vision architectures.
Summary updated