Deep Stochastic Configuration Networks with Universal Approximation Property

Dianhui Wang, Ming LiIJCNN 2018 · OpenReview

TL;DR

This paper introduces Deep Stochastic Configuration Networks (DeepSCNs), a novel method for incrementally building deep neural networks using a randomized approach. DeepSCNs offer a guaranteed universal approximation property and a fast learning algorithm, addressing key challenges in DNN design and implementation.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have shown immense promise, but their practical application is hindered by challenges in designing effective architectures and achieving fast, reliable training. Manually configuring DNNs is often a time-consuming trial-and-error process, and existing methods for automatic architecture determination may still require pre-defined depth. Furthermore, traditional training methods like backpropagation can suffer from issues like local minima and sensitivity to learning rate. This paper aims to provide a more systematic and efficient way to build DNNs that are both powerful and computationally tractable.

Methods

The core of this paper is the introduction of Deep Stochastic Configuration Networks (DeepSCNs). These networks are built incrementally, layer by layer. The key innovation lies in a randomized approach to configuring the weights and biases of hidden nodes within each layer, guided by a supervisory mechanism that constrains these random assignments. Crucially, all hidden layers have direct links to the output layer.

The learning process involves:

  1. Incremental Construction: Starting with a small network, DeepSCNs add hidden nodes and layers iteratively until a termination criterion (e.g., desired error tolerance) is met.
  2. Randomized Configuration: Hidden layer parameters (weights and biases) are randomly assigned, but within specific constraints defined by the supervisory mechanism.
  3. Least Squares for Read-out Weights: The weights connecting each hidden layer directly to the output layer are efficiently learned using the least squares method.
  4. Universal Approximation Property: A theoretical result is established proving that DeepSCNs possess the universal approximation property, meaning they can approximate any continuous function.

The paper also presents a learning algorithm (Algorithm 1) that details this constructive process, allowing for either specific architecture definition or self-organization.

Datasets

The paper uses synthetic data for function approximation and a collection of handwritten digits for a case study. Specifically, for the case study on rotation angle prediction, they use 5000 training and 5000 test images of digits.

Key Insights and Ideas

  • Randomization with Constraints: The paper demonstrates that carefully constrained randomization of hidden layer parameters can lead to powerful and theoretically sound deep learning models. This offers an alternative to purely gradient-based optimization.
  • Direct Hidden-to-Output Links: The architecture with direct connections from every hidden layer to the output layer allows for a more flexible and multi-resolution learning representation, unlike standard deep networks that rely solely on the final hidden layer.
  • Guaranteed Universal Approximation: Establishing the universal approximation property for DeepSCNs provides a strong theoretical foundation, assuring their capacity to learn complex functions.
  • Fast and Incremental Learning: The constructive, layer-by-layer approach with least squares for read-out weights enables rapid model building, addressing a significant bottleneck in deep learning.
  • Data-Dependent Representation: The generated basis functions are data-dependent, leading to a "learning representation" that captures essential features of the input data.

Key Results

  • Theoretical Guarantee: DeepSCNs are proven to have the universal approximation property.
  • Empirical Performance: In a function approximation task, DeepSCNs demonstrated faster convergence and better approximation accuracy compared to standard SCNs.
  • Case Study (Handwritten Digits): For rotation angle prediction, DeepSCNs achieved higher Percentage of Predictions within an Acceptable error margin (PPA) and lower Root-Mean-Square Error (RMSE) compared to SCNs. For example, with specific architectures, DSCN achieved 65.21% PPA and 7.0265 RMSE, while SCN achieved 60.35% PPA and 7.5623 RMSE.
  • Algebraic Property: DeepSCNs exhibit less rank-deficiency in their hidden output matrix compared to SCNs, suggesting a more efficient and less redundant set of basis functions.
  • Robustness: The algorithm showed robustness to variations in the learning parameter 'r'.

Limitations and Future Work

  • Interpretability: While DeepSCNs are effective, their learning representation is noted as being "less interpretable."
  • Theoretical Justification for Consistency: The paper mentions that the observed consistency between learning and generalization is based on experimental observations, lacking theoretical justification.
  • Future Work: The authors suggest exploring algebraic properties of DeepSCNs for quality evaluation, developing new supervisory mechanisms, advanced learning schemes for random representation, and investigating convolutional or recurrent versions of DeepSCNs. They also highlight the potential for extracting random features for big data classification with hierarchical or cascade structures.

Summary updated

Abstract

This paper develops a randomized approach for incrementally building deep neural networks, where a superviso- ry mechanism is proposed to constrain the random assignment of the weights and biases, and all the hidden layers have direct links to the output layer. A fundamental result on the universal approximation property is established for such a class of randomized leaner models, namely deep stochastic configuration networks (DeepSCNs) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> . A learning algorithm is presented to implement DeepSCNs with either specific architecture or self- organization. The read-out weights attached with all direct links from each hidden layer to the output layer are evaluated by the least squares method. Given a set of training examples, DeepSCNs can speedily produce a learning representation, that is, a collection of random basis functions with the cascaded inputs together with the read-out weights. An empirical demonstration on a function approximation and a case study are carried out to demonstrate some properties of the proposed deep learner model.
BibTeX
@inproceedings{wang2018deep,
  author = {Dianhui Wang and Ming Li},
  title = {{Deep Stochastic Configuration Networks with Universal Approximation Property}},
  booktitle = {IJCNN},
  year = {2018},
  url = {https://openreview.net/forum?id=XzGnRfM6OXh},
}
Figure