SGB-ELM: An Advanced Stochastic Gradient Boosting-Based Ensemble Scheme for Extreme Learning Machine

Hua Guo, Jikui Wang, Wei Ao, Yulin HeComput. Intell. Neurosci 2018 · DOI

TL;DR

This paper introduces SGB-ELM, a novel ensemble method that combines Extreme Learning Machines (ELMs) with stochastic gradient boosting. By optimizing a regularized objective function using a second-order approximation and iteratively updating weights, SGB-ELM achieves improved generalization performance and robustness compared to existing ELM ensemble techniques.

What Problem Does This Paper Address?

Extreme Learning Machines (ELMs) are known for their fast training speeds due to random initialization of input weights and biases, followed by analytical determination of output weights. However, this random initialization can lead to suboptimal parameters, negatively impacting generalization and robustness. While ensemble methods have been proposed to address this, this paper aims to develop a more advanced ELM ensemble scheme that specifically leverages the strengths of stochastic gradient boosting to further enhance performance and mitigate overfitting.

Methods

The core of this work is the Stochastic Gradient Boosting-Based Extreme Learning Machine (SGB-ELM) algorithm. Unlike traditional boosting methods that might directly integrate ELMs, SGB-ELM constructs an ensemble by sequentially training weak ELM learners. The key innovations are:

  • Additive Training with Regularized Objective: Each new ELM is trained additively to improve the ensemble. A novel objective function is designed that includes a regularization term to combat overfitting.
  • Second-Order Optimization: The output weights of each weak ELM are determined by optimizing this objective using a second-order approximation. This provides a derivation formula for updating the weights.
  • Iterative Weight Update: Since the derivation formula is hard to solve analytically, the algorithm uses the output weights learned from pseudo-residuals as an initial heuristic. These weights are then iteratively updated using the derivation formula.
  • Stochastic Subset Selection: To further improve robustness and avoid overfitting, SGB-ELM uses a randomly selected subset of the training data at each iteration, rather than the entire dataset.
  • Initial Base Learner Enhancement: To control the fitting ability of the initial base learner and reduce instability from random parameter initialization, SGB-ELM performs multiple random initializations and averages the results.

Datasets

The paper evaluates SGB-ELM on:

  • 4 KEEL regression datasets: Laser, Friedman, Mortgage, and Wizmir.
  • 5 UCI classification datasets: Image segmentation, Texture, Spambase, Banana, and Ring.

These are standard benchmark datasets commonly used in machine learning research. No new benchmarks were introduced.

Key Insights and Ideas

  • The power of regularization in boosting ELMs: The inclusion of a regularization term in the objective function is crucial for preventing overfitting, a common issue in boosting methods.
  • Second-order optimization for efficient weight updates: Using second-order approximation provides a more robust and potentially faster convergence path for determining the optimal weights of individual ELMs.
  • Stochasticity as a double-edged sword: While using random subsets enhances diversity and reduces overfitting, it can also slow down convergence. The paper empirically investigates the impact of subset size.
  • Initial base learner quality matters: The paper highlights that a too-strong initial base learner can negatively impact the ensemble's effectiveness. Their method of averaging multiple random initializations helps to create a more balanced starting point.

Key Results

  • Improved Generalization: SGB-ELM consistently achieved better testing RMSE for regression tasks and higher testing accuracy for classification tasks compared to baseline ELM, simple ensemble ELM, Bagging ELM, and Adaboost ELM.
  • Robustness and Stability: The proposed method demonstrated better predicted robustness and stability, often with lower variance in performance across trials.
  • Competitive Training Time: While SGB-ELM's training time is higher than basic ELM, it is competitive with or better than other ensemble methods like Adaboost, especially considering its performance gains.
  • Parameter Sensitivity: The paper shows that the number of hidden nodes (L) and the regularization factor (λ) significantly impact performance, with a proper balance needed to avoid overfitting. The optimal sampling fraction (Ñ/N) also varies by dataset.

Limitations and Future Work

  • Parameter Tuning: The performance of SGB-ELM is sensitive to its learning parameters (L, λ, Ñ/N), requiring careful tuning, which can be computationally intensive. The paper provides empirical guidance but not a definitive automatic selection method.
  • Computational Cost: While competitive, the ensemble nature and iterative optimization inherently lead to higher training times than a single ELM.
  • Future Work: The paper suggests exploring different types of loss functions and regularization techniques, as well as investigating the impact of different activation functions for the ELM base learners. Further research into more efficient parameter selection strategies could also be beneficial.

Summary updated

BibTeX
@article{guo2018sgb,
  author = {Hua Guo and Jikui Wang and Wei Ao and Yulin He},
  title = {{SGB-ELM: An Advanced Stochastic Gradient Boosting-Based Ensemble Scheme for Extreme Learning Machine}},
  journal = {Comput. Intell. Neurosci},
  year = {2018},
  doi = {10.1155/2018/4058403},
}
Figure