Abstract
Enhancing the generalization performance of neural networks given limited data availability remains a formidable challenge, due to the model selection trade-off between training error and generalization gap. To handle this challenge, we present a posterior optimization issue, specifically designed to reduce the generalization error of trained neural networks. To operationalize this concept, we propose a Doubly-Robust Boosting machine (DRBoost) which consists of a statistical learner and a zero-order optimizer. The statistical learner reduces the model capacity and thus the generalization gap; the zero-order optimizer minimizes the training error in a gradient-free manner. The two components cooperate to reduce the generalization error of a fully trained neural network in a doubly robust manner. Furthermore, the statistical learner alleviates the multicollinearity in the discriminative layer and enhances the generalization performance. The zero-order optimizer eliminates the reliance on gradient calculation and offers more flexibility in learning objective selection. Experiments demonstrate that DRBoost improves the generalization performance of various prevalent neural network backbones effectively.
Introduction
Deep learning models typically require massive datasets to generalize well. However, in many real-world regression scenarios—such as traffic forecasting or energy grid management—data can be scarce. In these "data-limited" regimes, standard training via Stochastic Gradient Descent (SGD) faces significant hurdles: unstable estimates due to multicollinearity, a "surrogate gap" between differentiable loss functions and actual performance metrics, and a punishing trade-off between training error and the generalization gap.
To address these issues, we introduce DRBoost (Doubly-Robust Boosting), a "post-optimization" framework designed to refine pre-trained neural networks. Instead of replacing SGD, DRBoost acts as a plugin that treats the SGD-trained model as a warm start, using specialized tools to overcome the bottlenecks of traditional backpropagation.
The DRBoost Framework
DRBoost decomposes the generalization error into two components: the training error ($E_{in}$) and the generalization gap ($E_{out} - E_{in}$). By attacking both sides of this equation, the framework achieves a "doubly robust" improvement in model performance.
As illustrated in Figure 1, the framework consists of two main pillars:
- The Statistical Learner ($A_s$): The final discriminative layer of the network is replaced with a statistical model, specifically Partial Least Squares Regression (PLSR). PLSR projects inputs into a lower-dimensional space, reducing the model's complexity (VC dimension). This effectively shrinks the generalization gap and handles multicollinearity by design.
- The Zero-Order Optimizer ($A_z$): Because the statistical learner makes gradient backpropagation difficult, we employ metaheuristic optimizers like Particle Swarm Optimization (PSO). This gradient-free approach tunes the weights of the penultimate layer, allowing the model to optimize directly for the target metric and escape local minima that often trap SGD.
Implementation and Workflow
The DRBoost workflow begins with a network backbone pre-trained via standard SGD. Once initial training is complete, the early layers are frozen. The Zero-Order Optimizer then tunes the penultimate layer while the Statistical Learner fits the final output based on latent features.
Experimental Results
We evaluated DRBoost across five real-world time-series regression benchmarks, including traffic speed (METR-LA, PEMSBAY) and electricity consumption. These datasets represent complex, high-dimensional tasks where data is relatively limited compared to model size.
Key findings include:
- Consistent Performance Gains: DRBoost improved the $R^2$ and lowered the Mean Absolute Error (MAE) across all tested backbones (MLPs, LSTMs, TCNs, and GraphWaveNet). For example, on the TRAFFIC dataset, a base MLP's $R^2$ improved from 0.835 to 0.853.
- Overfitting Mitigation: The framework proved highly effective at calibrating complex models. GraphWaveNet, which initially struggled on the PEMS04 dataset, saw its $R^2$ rise to 0.868 after DRBoost refinement.
- Direct Metric Optimization: By bypassing the surrogate gap, the zero-order optimizer allows for the direct optimization of non-differentiable metrics that are typically inaccessible during standard training.
Conclusion
DRBoost provides a flexible and robust paradigm for improving neural network generalization in data-scarce environments. By combining the complexity-reduction strengths of statistical learning with the global search capabilities of metaheuristic optimization, it offers a powerful tool for refining high-stakes regression models. Future research will explore the application of DRBoost in other low-data domains, such as drug discovery and reinforcement learning.
Summary updated