Abstract
For various optimization methods, gradient descent-based algorithms can achieve outstanding performance and have been widely used in various tasks. Among those commonly used algorithms, ADAM owns many advantages such as fast convergence with both the momentum term and the adaptive learning rate. However, since the loss functions of most deep neural networks are non-convex, ADAM also shares the drawback of getting stuck in local optima easily. To resolve such a problem, the idea of combining genetic algorithm with base learners is introduced to rediscover the best solutions. Nonetheless, from our analysis, the idea of combining genetic algorithm with a batch of base learners still has its shortcomings. The effectiveness of genetic algorithm can hardly be guaranteed if the unit models converge to close or the same solutions. To resolve this problem and further maximize the advantages of genetic algorithm with base learners, we propose to implement the boosting strategy for input model training, which can subsequently improve the effectiveness of genetic algorithm. In this paper, we introduce a novel optimization algorithm, namely Boosting based Genetic ADAM (BGADAM). With both theoretic analysis and empirical experiments, we will show that adding the boosting strategy into the BGADAM model can help models jump out the local optima and converge to better solutions.
Introduction
Deep learning models rely on optimization to find the global minimum of a loss function. While the ADAM optimizer is a popular choice due to its speed, it often gets trapped in local optima because neural network loss landscapes are highly non-convex. Previous attempts to solve this, like GADAM, integrated Genetic Algorithms (GA) to treat model parameters as "genes" and use crossover and mutation to "jump" out of local traps. However, these methods suffer from "local sticking": if the initial parent models are too similar, their offspring will likely remain stuck in the same local neighborhood, rendering the evolutionary process ineffective.
BGADAM (Boosting-based Genetic ADAM) addresses this by treating diversity as a prerequisite for evolution. By forcing "disagreement" between models through a boosting strategy, the optimization process avoids narrowing its focus too early, effectively using diversity as a mechanism to explore the parameter space more thoroughly.
Methodology
The BGADAM framework integrates a boosting strategy into the Genetic-ADAM hybrid to ensure population diversity. The process operates over $K$ generations:
- Boosting-based Training: Instead of training multiple models on identical data, BGADAM re-weights the training dataset. Subsequent models focus on samples misclassified by previous ones, ensuring that the $g$ models in each generation explore different regions of the parameter space.
- Fitness Evaluation: Models are ranked based on their validation loss.
- Genetic Operations: High-performing "parent" models undergo crossover (exchanging parameter segments) and mutation (random perturbations) to generate "child" models.
- Selection: The best models from the combined pool of parents and children are selected to seed the next generation.
The key innovation is using boosting as a diversity engine. By forcing models to converge to different local optima, the subsequent Genetic Algorithm has a significantly higher probability of discovering a superior path toward the global optimum.
Results and Discussion
The researchers evaluated BGADAM on three standard computer vision benchmarks: the ORL face recognition dataset, MNIST handwritten digits, and CIFAR-10 color images.
BGADAM consistently outperformed standard ADAM, GADAM, and other popular optimizers like RMSProp and AdaGrad. On CIFAR-10, BGADAM achieved a test accuracy of 0.6358, surpassing GADAM (0.6189) and ADAM (0.6103). The most significant improvement was observed in test loss reduction. On the ORL dataset, BGADAM achieved a loss of 0.0947, roughly 43% lower than GADAM's 0.1658.
Empirical results suggest that the algorithm typically converges to an ideal solution within approximately 5 generations ($K=5$) with a population size of 5 models ($g=5$).
Conclusion
BGADAM demonstrates that combining boosting with genetic algorithms can effectively overcome the local optima problem inherent in gradient-based optimizers like ADAM. While the approach introduces computational overhead by training multiple models, it provides a more robust search mechanism for non-convex landscapes. Future work will focus on scaling this evolutionary boosting approach to larger architectures like Transformers and ResNets and optimizing the hyperparameter tuning process for $g$ and $K$.
Summary updated