Ensemble Neural Networks (ENN): A gradient-free stochastic method

Yuntian Chen, Haibin Chang, Jin Meng, Dongxiao ZhangNeural Networks 2019 · DOI · OpenReview

TL;DR

This paper introduces Ensemble Neural Networks (ENN), a novel gradient-free stochastic method for neural network training. ENN leverages the Ensemble Randomized Maximum Likelihood (EnRML) algorithm to perform uncertainty quantification and robustly handle small datasets, making it particularly valuable for real-world engineering applications where data is scarce and expensive.

What Problem Does This Paper Address?

Traditional neural networks often rely on gradient-based optimization methods, which can be problematic when dealing with complex, non-differentiable neuron models or loss functions. Furthermore, many real-world engineering problems suffer from limited data availability, making it challenging to train deep neural networks effectively. Additionally, quantifying the uncertainty associated with predictions is crucial for safety-critical applications, but existing methods can be computationally expensive or struggle with small datasets. This paper aims to bridge these gaps by proposing a method that is gradient-free, robust to small data sizes, and capable of performing uncertainty quantification.

Methods

The core innovation is the Ensemble Neural Network (ENN) which integrates the Ensemble Randomized Maximum Likelihood (EnRML) algorithm into the neural network training process. Instead of relying on derivatives for optimization, ENN uses covariance matrices derived from stochastic realizations of model parameters. The EnRML algorithm, originally developed for history matching in petroleum engineering, is adapted to optimize neural network weights. This approach allows for:

  • Gradient-Free Optimization: Eliminates the need for calculating derivatives, enabling the use of more complex neuron models and loss functions.
  • Uncertainty Quantification: Built on a Bayesian framework, ENN naturally provides probability distributions for weights and predictions, allowing for confidence estimation.
  • Robustness to Small Datasets: The ensemble of stochastic realizations effectively enlarges the training dataset, improving performance even with limited data.
  • Shared Information: Unlike traditional Monte Carlo methods where realizations are independent, ENN's realizations share information through covariance matrices, simplifying the backward process and improving convergence.

Datasets

The paper evaluates ENN using several datasets:

  • Toy Regression Dataset: A one-dimensional dataset proposed by Hernandez-Lobato and Adams (2015) to assess uncertainty quantification capabilities.
  • Highly Nonlinear Ideal Dataset: A synthetic dataset designed as a sanity check for the ENN's functionality and to study the influence of network architecture, data size, and observation errors.
  • Real-World Datasets: Three datasets from the UCI Repository of Machine Learning Databases (Auto MPG, Combined Cycle Power Plant, and Concrete Slump) and a highly nonlinear petroleum engineering dataset (PRES-2D) are used for generalization tests.

Key Insights and Ideas

  • Covariance is King: The ENN demonstrates that covariance information from stochastic realizations can be as powerful as gradients for optimization, opening new avenues for training complex models.
  • Ensemble Power for Small Data: The ensemble approach doesn't just average; it actively uses the statistical properties of multiple realizations to overcome the limitations of scarce training data, akin to data augmentation.
  • Bayesian Foundation for Practicality: By grounding ENN in a Bayesian framework, the method naturally provides uncertainty estimates, a critical feature often missing or complex to implement in other neural network approaches.
  • Gradient-Free Advantage: The ability to avoid derivatives is a significant advantage for emerging areas like brain-inspired computing with complex neuron models (e.g., Hodgkin-Huxley) or natural language processing tasks with non-differentiable evaluation metrics.

Key Results

  • Improved Performance on Small Datasets: Experiments show ENN significantly outperforms traditional Bayesian Neural Networks (BNN) when training data is scarce, achieving higher prediction accuracy. For instance, when the ratio of training data to model parameters is small, ENN shows a clear advantage (Fig. 8).
  • Robustness to Measurement Error: ENN demonstrates better robustness to varying scales of measurement errors compared to BNN, maintaining performance even with noisy observations (Fig. 9).
  • Comparable Performance with Sufficient Data: When sufficient training data is available, ENN's performance approaches that of BNN, and in some cases, BNN might even show slightly lower loss, likely due to the artificial noise added by ENN in its small-data strategy.
  • Successful Generalization: ENN achieves competitive or superior results across various real-world datasets compared to FCNN and different BNN variants, particularly in scenarios with limited data. For example, in the PRES-2D dataset, ENN consistently shows lower estimation loss than BNN methods.

Limitations and Future Work

  • Computational Cost: While the EnRML algorithm is designed to reduce computational complexity, especially for small datasets, it can still be computationally intensive, particularly for very large models.
  • Hyperparameter Tuning: Although ENN's hyperparameters (covariance matrices) are determined from prior information, their accurate estimation is crucial for optimal performance.
  • Future Work: The authors suggest that ENN can be conveniently formulated for sequential (online) learning and that the embedded EnRML is embarrassingly parallel, indicating potential for further scalability and application in real-time systems. The gradient-free nature also makes it suitable for more complex neuron models and applications like natural language processing.

Summary updated

Abstract

Highlights•Developing a stochastic gradient-free neural network based on ensemble randomized maximum likelihood.•Utilizing covariance matrices for optimization instead of gradients in the training process.•Able to simultaneously provide estimations and perform uncertainty quantification.•Robust to small data size and suitable for real-world engineering applications.
BibTeX
@article{chen2019ensemble,
  author = {Yuntian Chen and Haibin Chang and Jin Meng and Dongxiao Zhang},
  title = {{Ensemble Neural Networks (ENN): A gradient-free stochastic method}},
  journal = {Neural Networks},
  year = {2019},
  doi = {10.1016/j.neunet.2018.11.009},
}
Figure