Deep Stochastic Configuration Networks: Universal Approximation and Learning Representation

Dianhui Wang, Ming LiarXiv 2017 · arXiv

Introduction

Designing deep neural networks (DNNs) is often considered a "dark art," requiring significant trial-and-error to determine optimal layer counts and node configurations. Standard training via backpropagation, while powerful, is computationally expensive and prone to issues like local minima and vanishing gradients. This paper introduces Deep Stochastic Configuration Networks (DeepSCNs), a framework that bridges the gap between fast randomized learning and the representational power of deep architectures.

Methodology

The core of the DeepSCN algorithm is an incremental construction process that builds the network from the ground up, ensuring universal approximation through a mathematically rigorous "supervised" randomized approach.

Incremental Construction and Supervised Randomness

Unlike standard DNNs, a DeepSCN starts with a single node and adds nodes one by one. Once a layer reaches a predefined limit or error tolerance, the algorithm begins a new hidden layer. To ensure each addition improves the model, parameters (weights and biases) are generated randomly but only accepted if they satisfy a specific inequality constraint related to the current residual error. This supervisory mechanism guarantees that every new node contributes to error reduction.

Multi-Resolution Architecture

A defining feature of DeepSCNs is that every hidden layer maintains direct links to the output layer. This allows the model to capture "multi-resolution" features, utilizing both coarse and fine details of the data simultaneously.

Architecture of a DeepSCN showing direct links from all hidden layers to the output layer.

Because the hidden parameters are fixed once assigned, the "read-out" weights (the links to the output) can be calculated instantly using the least squares method. This avoids iterative gradient descent, leading to significantly faster training times.

Key Insights

  • Reduced Redundancy: Rank-deficiency analysis demonstrates that the deep structure of DeepSCNs produces less redundant basis functions than shallow SCNs, making the model more efficient with its neurons.
  • Fast Convergence: By solving output weights via a closed-form linear equation, the model reaches high accuracy much faster than models requiring thousands of epochs of backpropagation.
  • Robustness: The supervisory mechanism effectively tames the inherent randomness of the weight initialization, providing high stability across different random seeds.

Experimental Results

The authors evaluated DeepSCNs using synthetic function approximation and a handwritten digit rotation regression task. In the digit rotation case study, DeepSCN consistently outperformed shallow SCNs across all metrics.

Comparison of Training and Test RMSE between DSCN and shallow SCN as the number of hidden nodes increases.

As shown in the performance plots, the DeepSCN (DSCN) achieves a lower Root Mean Square Error (RMSE) more rapidly than the shallow version. For instance, one DeepSCN configuration achieved a 66.36% PPA (Percentage of Predictions within an Acceptable margin) compared to 60.00% for the shallow version, with a test RMSE of 12.09 versus 14.77.

Conclusion and Future Work

DeepSCNs provide a predictable, constructive alternative to "black box" deep learning. While the paper establishes a foundation for universal approximation and efficient learning, future research will focus on automating hyperparameter selection (such as nodes per layer) and extending the framework to convolutional and recurrent architectures for complex computer vision and sequence tasks.

Summary updated

Abstract

This paper develops a randomized approach for incrementally building deep neural networks, where a supervisory mechanism is proposed to constrain the random assignment of the weights and biases, and all the hidden layers have direct links to the output layer. A fundamental result on the universal approximation property is established for such a class of randomized leaner models, namely deep stochastic configuration networks (DeepSCNs). A learning algorithm is presented to implement DeepSCNs with either specific architecture or self-organization. The read-out weights attached with all direct links from each hidden layer to the output layer are evaluated by the least squares method. Given a set of training examples, DeepSCNs can speedily produce a learning representation, that is, a collection of random basis functions with the cascaded inputs together with the read-out weights. An empirical study on a function approximation is carried out to demonstrate some properties of the proposed deep learner model.
BibTeX
@misc{wang2017deep,
  author = {Dianhui Wang and Ming Li},
  title = {{Deep Stochastic Configuration Networks: Universal Approximation and Learning Representation}},
  year = {2017},
  eprint = {1702.05639},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1702.05639},
}
Figure