Optimizing Deep Learning Models with Improved BWO for TEC Prediction

Yi Chen, Haijun Liu, Weifeng Shan, Yuan Yao, Lili Xing, Haoran Wang, Kunpeng ZhangBiomimetics 2024 · DOI

TL;DR

This paper tackles the challenge of optimizing deep learning models for predicting ionospheric Total Electron Content (TEC) by proposing an enhanced Beluga Whale Optimization (BWO) algorithm called FAMBWO. FAMBWO significantly improves upon the original BWO's ability to find optimal hyperparameters, leading to a more accurate and robust TEC prediction model (MA-BiLSTM).

What Problem Does This Paper Address?

Accurately predicting the Total Electron Content (TEC) of the ionosphere is crucial for applications like GPS navigation, space weather monitoring, and wireless communication. While deep learning models have shown great promise in this area, their performance is heavily reliant on a large number of hyperparameters (e.g., learning rate, batch size, number of hidden layers). Finding the optimal combination of these hyperparameters – a process known as hyperparameter optimization (HPO) – is a computationally expensive and challenging task, often leading to suboptimal model performance. Existing optimization methods like grid search and random search are inefficient, while swarm intelligence algorithms, though promising, can get stuck in local minima. This paper aims to address these limitations by developing a more effective HPO algorithm and applying it to TEC prediction.

Methods

The core of this paper lies in two main contributions:

  1. Improved Beluga Whale Optimization (FAMBWO): The authors identify two key drawbacks in the original Beluga Whale Optimization (BWO) algorithm: insufficient initial population diversity and an imbalance between exploration and exploitation phases, leading to local optima. To address these, they introduce three strategies:
  2. Cat Chaotic Mapping (CCM): Used for initializing the population to enhance diversity.
  3. Firefly Algorithm (FA) Strategy: Incorporated into the exploration phase to improve global search capabilities.
  4. Cauchy Mutation and Tent Chaotic Mapping (CMT) Strategy: Integrated into the exploitation phase to help escape local minima and optimize nonlinear functions. The combination of these strategies results in the proposed FAMBWO algorithm.

  5. FAMBWO-MA-BiLSTM Framework: The authors propose a novel deep learning model for TEC prediction called Multi-head Attentional Bidirectional Long Short-Term Memory (MA-BiLSTM). This model incorporates multi-head attention to capture complex temporal dependencies. Crucially, they then use their improved FAMBWO algorithm to optimize the key hyperparameters of the MA-BiLSTM model (number of BiLSTM units, dropout ratio, learning rate, and batch size).

Datasets

The study utilizes TEC data provided by the Center for Orbit Determination in Europe (CODE). The dataset spans from January 1, 1999, to April 30, 2015, containing approximately 77,467 TEC values. The data undergoes preprocessing, including a first-order difference to ensure stationarity and min-max normalization to scale the data. The dataset is split into training (first 14 years) and testing (remaining years) sets.

Key Insights and Ideas

  • HPO is Paramount: The paper strongly emphasizes that effective hyperparameter optimization can significantly boost model performance, often more so than model selection alone.
  • Synergy of Strategies: Combining CCM, FA, and CMT strategies within BWO leads to a synergistic effect, resulting in a superior optimization algorithm (FAMBWO) that outperforms its individual components and the original BWO.
  • Targeted Improvements for BWO: The identified weaknesses of BWO (population diversity and exploration/exploitation balance) are directly addressed by the proposed strategies, leading to a more robust and efficient optimizer.
  • MA-BiLSTM's Potential: The proposed MA-BiLSTM model, with its multi-head attention mechanism, shows promise for capturing complex temporal patterns in TEC data.

Key Results

  • FAMBWO Performance on Benchmarks: FAMBWO demonstrated superior performance on 30 benchmark functions compared to 11 other state-of-the-art swarm intelligence algorithms, exhibiting faster convergence and better solutions. It ranked first in exploration, exploitation, and local optimal avoidance abilities.
  • FAMBWO-MA-BiLSTM for TEC Prediction: The FAMBWO-MA-BiLSTM framework significantly outperformed MA-BiLSTM models optimized by grid search, random search, and Bayesian optimization. Specifically, it achieved:
  • Reduced MSE by 5.98% compared to BWO-MA-BiLSTM.
  • Reduced RMSE by 3.03% compared to BWO-MA-BiLSTM.
  • Reduced MAE by 4.37% compared to BWO-MA-BiLSTM.
  • Improved R² value to 0.9803, indicating a stronger correlation between predicted and true TEC values.

Limitations and Future Work

  • Limitations of BWO: The paper acknowledges that the original BWO algorithm suffers from insufficient population diversity and an imbalanced exploration-exploitation phase, leading to local optima. While FAMBWO addresses these, the inherent complexity of HPO remains a challenge.
  • Future Work: The authors suggest that their proposed framework provides a new solution for deep learning hyperparameter optimization in TEC prediction and can serve as a reference for similar optimization tasks in other deep learning applications. Further research could explore applying FAMBWO to a wider range of deep learning models and different scientific domains.

Summary updated

Abstract

The prediction of total ionospheric electron content (TEC) is of great significance for space weather monitoring and wireless communication. Recently, deep learning models have become increasingly popular in TEC prediction. However, these deep learning models usually contain a large number of hyperparameters. Finding the optimal hyperparameters (also known as hyperparameter optimization) is currently a great challenge, directly affecting the predictive performance of the deep learning models. The Beluga Whale Optimization (BWO) algorithm is a swarm intelligence optimization algorithm that can be used to optimize hyperparameters of deep learning models. However, it is easy to fall into local minima. This paper analyzed the drawbacks of BWO and proposed an improved BWO algorithm, named FAMBWO (Firefly Assisted Multi-strategy Beluga Whale Optimization). Our proposed FAMBWO was compared with 11 state-of-the-art swarm intelligence optimization algorithms on 30 benchmark functions, and the results showed that our improved algorithm had faster convergence speed and better solutions on almost all benchmark functions. Then we proposed an automated machine learning framework FAMBWO-MA-BiLSTM for TEC prediction, where MA-BiLSTM is for TEC prediction and FAMBWO for hyperparameters optimization. We compared it with grid search, random search, Bayesian optimization algorithm and beluga whale optimization algorithm. Results showed that the MA-BiLSTM model optimized by FAMBWO is significantly better than the MA-BiLSTM model optimized by grid search, random search, Bayesian optimization algorithm, and BWO.
BibTeX
@inproceedings{chen2024optimizing,
  author = {Yi Chen and Haijun Liu and Weifeng Shan and Yuan Yao and Lili Xing and Haoran Wang and Kunpeng Zhang},
  title = {{Optimizing Deep Learning Models with Improved BWO for TEC Prediction}},
  booktitle = {Biomimetics},
  year = {2024},
  doi = {10.3390/biomimetics9090575},
}
Figure