A Novel Hybrid Grid Search and Tree Parzen Estimator for Deep Learning Hyperparameters Optimization

Souhir Khessiba, Ahmed Ghazi Blaiech, Asma Ben Abdallah, Antoine Manzanera, Khaled Ben Khalifa, Mohamed Hédi BedouiAICCSA 2024 · DOI

TL;DR

This paper introduces GS-TPE, a novel hybrid approach for optimizing hyperparameters in deep learning models, specifically for EEG-based vigilance state classification. By combining the exhaustive search of Grid Search with the intelligent exploration of Tree Parzen Estimator, GS-TPE significantly improves classification accuracy compared to individual methods and existing baselines.

What Problem Does This Paper Address?

Optimizing hyperparameters is a critical but computationally expensive challenge in deep learning, especially for complex tasks like medical signal analysis. Deep learning models have many hyperparameters that heavily influence their performance, and manual tuning is impractical. Existing automated methods like Grid Search (GS) can be inefficient due to the vast search space, while others like Tree Parzen Estimator (TPE) can benefit from a more informed starting point. This paper aims to bridge this gap by developing a hybrid approach that leverages the strengths of both GS and TPE to achieve better and more efficient hyperparameter optimization for vigilance state classification using EEG signals.

Methods

The core of this paper is the proposed GS-TPE (Grid Search-Tree Parzen Estimator) hybrid approach. The methodology involves a two-phase process:

  1. Coarse Grid Exploration (GS): Grid Search is first employed to explore a defined hyperparameter space. It systematically evaluates all possible combinations within predefined ranges. The top 5 best-performing configurations (based on validation accuracy) are then selected. This initial phase helps to narrow down the search space to promising regions.
  2. Intelligent Refinement (TPE): The 5 selected configurations from GS serve as initial starting points for the Tree Parzen Estimator (TPE) algorithm. TPE, an enhanced Bayesian optimization technique, then intelligently explores the hyperparameter space around these starting points. It models the probability distribution of good and bad configurations to efficiently find optimal values. This phase aims to fine-tune the hyperparameters for maximum performance.

The paper also details the deep learning architectures used:

  • LSTM (Long Short-Term Memory): A stacked four-layer LSTM network for processing temporal data.
  • 1D-CNN-LSTM: A hybrid architecture combining 1D Convolutional Neural Networks (CNNs) for feature extraction with LSTM layers for temporal dependency capture.
  • AE-LSTM: An architecture that uses an Auto-Encoder (AE) to learn a latent representation of EEG signals, which then serves as input to an LSTM network.

Datasets

The study utilizes EEG data collected from six healthy subjects. The data was recorded over multiple 24-hour sessions with 15-day intervals, capturing both vigilance state (VS) and drowsiness state (DS). The EEG signals were pre-processed by extracting features from 4-second segments, resulting in feature vectors of size 21. These sequences were then divided into training (60%), validation (20%), and test (20%) sets. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Synergy of GS and TPE: The hybrid GS-TPE approach effectively combines the broad coverage of Grid Search with the efficient, intelligent exploration of Tree Parzen Estimator. This synergy leads to superior performance compared to using either method alone.
  • Importance of Initial Guess: Using the top GS configurations as starting points for TPE significantly guides the optimization process, preventing TPE from getting stuck in suboptimal regions and accelerating convergence.
  • EEG Vigilance State Classification: The paper demonstrates that advanced hyperparameter optimization is crucial for achieving high accuracy in complex medical signal analysis tasks like EEG-based vigilance state detection.
  • Architecture Performance: The 1D-CNN-LSTM architecture generally shows strong performance, and the GS-TPE optimization further enhances its capabilities.

Key Results

The GS-TPE approach demonstrated significant improvements in vigilance state classification accuracy across all tested deep learning architectures:

  • LSTM: Achieved a test accuracy of 91.15% with GS-TPE, an improvement over GS (89.38%) and TPE (90.20%).
  • 1D-CNN-LSTM: Reached a remarkable test accuracy of 93.74% with GS-TPE, outperforming GS (92.38%) and TPE (93.12%).
  • AE-LSTM: Achieved a test accuracy of 93.53% with GS-TPE, surpassing GS (90.07%) and TPE (88.37%).

Table IV shows that GS-TPE consistently outperformed individual GS, RS, and TPE methods. Table V further illustrates that the TPE refinement step within GS-TPE systematically improved the accuracy obtained by the initial GS configurations. For the 1D-CNN-LSTM, the best configuration found by GS-TPE (config 3') achieved an accuracy of 97.48% on the validation set.

Limitations and Future Work

The authors acknowledge the following:

  • Scalability: The study was conducted on a limited number of subjects. Future work should investigate the scalability of the GS-TPE approach with more subjects and more complex DL architectures.
  • Generalizability: Further research could explore the adaptability and generalizability of the hybrid GS-TPE approach to other complex DL applications beyond EEG analysis.

The paper suggests that future research should focus on these areas to further validate and expand the utility of the proposed GS-TPE method.

Summary updated

BibTeX
@inproceedings{khessiba2024novel,
  author = {Souhir Khessiba and Ahmed Ghazi Blaiech and Asma Ben Abdallah and Antoine Manzanera and Khaled Ben Khalifa and Mohamed Hédi Bedoui},
  title = {{A Novel Hybrid Grid Search and Tree Parzen Estimator for Deep Learning Hyperparameters Optimization}},
  booktitle = {AICCSA},
  year = {2024},
  doi = {10.1109/aiccsa63423.2024.10912622},
}
Figure