A Tutorial on Deep Neural Networks for Intelligent Systems

Juan C. Cuevas‐Tello, Manuel Valenzuela-Rendón, Juan A. Nolazco‐Flores

Abstract

Developing Intelligent Systems involves artificial intelligence approaches including artificial neural networks. Here, we present a tutorial of Deep Neural Networks (DNNs), and some insights about the origin of the term "deep"; references to deep learning are also given. Restricted Boltzmann Machines, which are the core of DNNs, are discussed in detail. An example of a simple two-layer network, performing unsupervised learning for unlabeled data, is shown. Deep Belief Networks (DBNs), which are used to build networks with more than two layers, are also described. Moreover, examples for supervised learning with DNNs performing simple prediction and classification tasks, are presented and explained. This tutorial includes two intelligent pattern recognition applications: hand-written digits (benchmark known as MNIST) and speech recognition.

Introduction

In the mid-2010s, "Deep Learning" transitioned from an academic niche to a cornerstone of modern AI. However, the transition from simple probabilistic models to complex multi-layered architectures often remained opaque to many researchers. This tutorial addresses the gap between high-level theory and practical implementation, focusing on unsupervised feature learning—the ability of a machine to identify patterns in raw data without explicit labels. By utilizing a specific MATLAB/Octave toolbox, the authors provide a reproducible roadmap for building and training these intelligent systems.

The Building Blocks: Restricted Boltzmann Machines (RBMs)

The fundamental unit of these architectures is the Restricted Boltzmann Machine (RBM). An RBM is a bipartite graph consisting of a visible layer (input) and a hidden layer (features). As an energy-based model, the RBM's objective is to find a weight configuration that minimizes the "energy" of the input data.

Structure of a Restricted Boltzmann Machine (RBM) showing visible and hidden layers.

Training these units typically employs Contrastive Divergence (CD). Unlike standard backpropagation, CD uses Gibbs sampling to approximate how weights should be adjusted to represent the input distribution. Notably, neurons in these models are stochastic; they exhibit a probability of "firing" based on a sigmoid activation function, rather than producing a deterministic output.

Scaling Up: Deep Belief Networks (DBNs)

The "deep" in deep learning refers to the stacking of these modular units. In a Deep Belief Network (DBN), the hidden features learned by one RBM serve as the visible input for the next. This hierarchical structure allows the network to learn increasingly abstract representations—moving from simple edges to complex shapes and objects.

Stacking RBMs to form a Deep Belief Network (DBN) where hidden layers provide features for subsequent levels.

Once the stack is pre-trained in an unsupervised manner, a final output layer is added for classification. The entire system is then "fine-tuned" using traditional supervised learning (backpropagation) to optimize performance for specific tasks.

Experimental Validation

The tutorial validates these methods across several benchmarks:

  • MNIST: A DNN based on a DBN achieved an error rate of 1.25%, outperforming contemporary Support Vector Machines (1.4%).
  • XOR Problem: Demonstrates the necessity of multi-layer networks for non-linear classification.
  • Speech Recognition: Discusses extracting Mel-Frequency Cepstral Coefficients (MFCCs) for acoustic modeling.

A key insight is the power of unsupervised pre-training, which initializes weights in a favorable region of the parameter space before supervised fine-tuning begins.

Discussion and Future Directions

While powerful, these models are sensitive to hyperparameters like learning rates and iteration counts. Furthermore, the computational cost was significant; training on MNIST took approximately three days on a 64-core system in 2016. The authors identify GPUs and parallel processing (MPI) as essential for scaling these models to larger, more complex datasets. While the focus here is on RBMs and DBNs, these principles laid the groundwork for the Convolutional and Recurrent architectures that dominate the field today.

Summary updated

Abstract

Developing Intelligent Systems involves artificial intelligence approaches including artificial neural networks. Here, we present a tutorial of Deep Neural Networks (DNNs), and some insights about the origin of the term "deep"; references to deep learning are also given. Restricted Boltzmann Machines, which are the core of DNNs, are discussed in detail. An example of a simple two-layer network, performing unsupervised learning for unlabeled data, is shown. Deep Belief Networks (DBNs), which are used to build networks with more than two layers, are also described. Moreover, examples for supervised learning with DNNs performing simple prediction and classification tasks, are presented and explained. This tutorial includes two intelligent pattern recognition applications: hand- written digits (benchmark known as MNIST) and speech recognition.
BibTeX
@misc{cuevastello2016tutorial,
  author = {Juan C. Cuevas‐Tello and Manuel Valenzuela-Rendón and Juan A. Nolazco‐Flores},
  title = {{A Tutorial on Deep Neural Networks for Intelligent Systems}},
  year = {2016},
  eprint = {1603.07249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1603.07249},
}
Figure