Selecting the Optimal Hidden Layer of Extreme Learning Machine Using Multiple Kernel Learning

Wentao Zhao, Pan Li, Qiang Liu, Dan Liu, Xinwang LiuKSII Trans. Internet Inf. Syst 2018 · DOI

Introduction

Extreme Learning Machines (ELMs) have gained popularity due to their exceptional learning speed and efficiency. Unlike traditional neural networks that rely on iterative backpropagation, ELMs randomly assign hidden layer weights and analytically determine output weights. However, this efficiency comes with a significant challenge: the "Goldilocks" problem of hidden layer sizing. Selecting too few hidden nodes leads to underfitting, while too many nodes cause overfitting and excessive computational cost. Traditionally, finding the optimal number of nodes has been a manual "guessing game" or a computationally expensive search process.

The MHLK-ELM Framework

To address the architectural instability of ELMs, this paper introduces the Multi-Hidden-Layer-Kernel ELM (MHLK-ELM). This framework leverages Multiple Kernel Learning (MKL) to automatically determine the optimal linear combination of various hidden layer configurations. The approach shifts the problem from a discrete search for a single node count to a continuous optimization problem.

The MHLK-ELM process consists of three primary stages:

  1. Hidden-Layer-Kernel ELM (HLK-ELM): A base model is created by integrating standard kernel functions with the random feature mapping of a traditional ELM, enhancing the model's ability to capture complex non-linear relationships.
  2. Multi-Layer Framework: Instead of selecting a single node count, the system generates a bank of HLK-ELMs with varying hidden node counts (e.g., 200 to 800).
  3. MKL Optimization: Each configuration is treated as a separate "kernel." The model then learns combination coefficients ($\gamma$) to weight the effectiveness of each hidden layer size for the specific task.

Optimization Strategies: Sparse vs. Non-Sparse

The authors propose two variants of the algorithm based on how they handle the combination coefficients:

  • Sparse MHLK-ELM: Employs an $l_1$-norm constraint to "zero out" less effective hidden layer configurations. This effectively performs automated model selection by identifying the most relevant node counts.
  • Non-Sparse MHLK-ELM: Uses an $l_q$-norm ($q>1$) to retain information from all configurations. This version often provides better stability as even "sub-optimal" layers can contribute unique information.
Algorithm 1: The iterative optimization process for Sparse MHLK-ELM.

The iterative process shown in Algorithm 1 demonstrates how the model refines the kernel matrix $K$ and updates the weights $\alpha$ and $\gamma$ until convergence, ensuring an optimal balance between the different hidden layer sizes.

Experimental Evaluation

The MHLK-ELM was evaluated across seven diverse datasets, including classification tasks (Glass, Ecoli, Vowel) and regression tasks (Auto price, Machine CPU, Housing, Stock). To ensure robustness against random initialization, the researchers conducted 10 independent trials for each dataset using a 60/40 training-testing split.

Key Results

  • Superior Accuracy: In classification, MHLK-ELM consistently achieved higher accuracy and mean Average Precision (mAP) compared to standard ELMs and single-kernel ELMs.
  • Significant Error Reduction: In regression tasks, such as the "Stock" dataset, the Root Mean Square Error (RMSE) was markedly lower than baseline models.
  • Enhanced Stability: The model exhibited lower standard deviations across trials, indicating it is significantly less sensitive to the randomness inherent in standard ELM weight initialization.

Conclusion and Future Work

MHLK-ELM successfully automates the selection of hidden layer sizes by treating architecture as a kernel optimization problem. While the non-sparse version often yields the best stability, the sparse version provides a streamlined model selection process.

Future research may focus on simultaneously optimizing both the number of nodes and the base kernel type (e.g., Gaussian vs. Laplacian) within a single framework. Additionally, further investigation into different mathematical norms could reveal new ways to improve generalization on unseen data.

Summary updated

BibTeX
@article{zhao2018selecting,
  author = {Wentao Zhao and Pan Li and Qiang Liu and Dan Liu and Xinwang Liu},
  title = {{Selecting the Optimal Hidden Layer of Extreme Learning Machine Using Multiple Kernel Learning}},
  journal = {KSII Trans. Internet Inf. Syst.},
  year = {2018},
  doi = {10.3837/tiis.2018.12.009},
}
Figure