Deep Sequential Neural Network

Ludovic Denoyer, Patrick GallinariarXiv 2014 · OpenReview

Abstract: Neural Networks sequentially build high-level features through their successive layers. We propose here a new neural network model where each layer is associated with a set of candidate mappings. When an input is processed, at each layer, one mapping among these candidates is selected according to a sequential decision process. The resulting model is structured according to a DAG like architecture, so that a path from the root to a leaf node defines a sequence of transformations. Instead of considering global transformations, like in classical multilayer networks, this model allows us for learning a set of local transformations. It is thus able to process data with different characteristics through specific sequences of such local transformations, increasing the expression power of this model w.r.t a classical multilayered network. The learning algorithm is inspired from policy gradient techniques coming from the reinforcement learning domain and is used here instead of the classical back-propagation based gradient descent techniques. Experiments on different datasets show the relevance of this approach.

Introduction

In standard Deep Neural Networks (DNNs), data flows through a fixed sequence of layers where every input undergoes the same set of mathematical transformations. While effective for homogeneous data, this "global" approach can be inefficient when datasets are highly diverse or heterogeneous. In "Deep Sequential Neural Network," Ludovic Denoyer and Patrick Gallinari propose a model that frames feature extraction as a sequential decision process, allowing the network to tailor its internal processing path to the specific characteristics of each input.

Architecture and Methodology

The core of the Deep Sequential Neural Network (DSNN) is a Directed Acyclic Graph (DAG) of representation spaces. Unlike a standard pipeline, the DSNN offers multiple candidate mappings at each layer.

The DAG Structure

  1. Nodes and Edges: Each node in the graph represents a feature space. The edges between nodes are mapping functions ($f$), consisting of standard weights and biases.
  2. The Router: Each node contains a selection function ($p$). This function acts as a controller, analyzing the current representation of the input to calculate a probability distribution over which "child" node to visit next.
  3. The Trajectory: For any given input, the model starts at a root node and sequentially selects a path to a leaf node. The final prediction is derived from the representation reached at the end of this specific trajectory.
The DAG architecture of a DSNN, showing how an input can follow different paths (trajectories) through the network based on selection functions.

Optimization

Because the selection process is discrete, standard back-propagation cannot be applied directly to the routers. The authors employ a hybrid training strategy:

  • Mapping functions ($f$): Optimized using standard back-propagation.
  • Selection functions ($p$): Optimized using policy gradient techniques from reinforcement learning, where the "reward" is the negative of the final classification loss.

Experimental Evaluation

The authors evaluated DSNNs against standard NNs on several benchmarks, including UCI datasets, MNIST-Negative (where 50% of images are color-inverted), and synthetic Checkerboard datasets.

Handling Heterogeneity: MNIST-Negative

In the MNIST-Negative task, a standard linear NN failed to generalize, achieving only ~28% accuracy. In contrast, a DSNN with just two possible actions per layer achieved 88.4% accuracy. The model successfully learned to branch its logic: one path specialized in normal digits, while another handled inverted ones.

Expressive Power: Checkerboard Patterns

On synthetic 2D datasets with complex "checkerboard" class distributions, DSNNs demonstrated significantly higher expressive power. As the complexity increased to an $11 \times 11$ grid, the DSNN maintained a significant lead (0.697 accuracy vs. 0.5 for a standard NN).

Decision frontiers on a 7x7 checkerboard. DSNNs (right columns) create much more intricate and accurate boundaries compared to standard NNs (left column) by utilizing local transformations.

Discussion and Conclusion

DSNNs represent a generalization of feed-forward networks; a standard NN is simply a DSNN where every node has exactly one child. By allowing for "local" transformations, the model can achieve high accuracy with fewer parameters per path than a massive global model would require to cover all edge cases.

While the model shows great promise for disparate data distributions, it faces challenges such as potential parameter explosion if the branching factor is too high and increased computational overhead during training due to the need for Monte Carlo sampling in policy gradients. Nonetheless, the DSNN offers a compelling framework for "smarter" architectures that actively decide how to process information based on the input's unique features.

Summary updated

Abstract

Neural Networks sequentially build high-level features through their successive layers. We propose here a new neural network model where each layer is associated with a set of candidate mappings. When an input is processed, at each layer, one mapping among these candidates is selected according to a sequential decision process. The resulting model is structured according to a DAG like architecture, so that a path from the root to a leaf node defines a sequence of transformations. Instead of considering global transformations, like in classical multilayer networks, this model allows us for learning a set of local transformations. It is thus able to process data with different characteristics through specific sequences of such local transformations, increasing the expression power of this model w.r.t a classical multilayered network. The learning algorithm is inspired from policy gradient techniques coming from the reinforcement learning domain and is used here instead of the classical back-propagation based gradient descent techniques. Experiments on different datasets show the relevance of this approach.
BibTeX
@misc{denoyer2014deep,
  author = {Ludovic Denoyer and Patrick Gallinari},
  title = {{Deep Sequential Neural Network}},
  year = {2014},
  eprint = {1410.0510},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1410.0510},
}
Figure