Neural Network based on Automatic Differentiation Transformation of Numeric Iterate-to-Fixedpoint

Mansura Habiba, Barak A. Pearlmutter

Abstract

This paper introduces the Temporal Wormhole Neural Network (TWNN), a novel architecture that replaces fixed-depth layers with a dynamic iterate-to-fixed-point mechanism. By leveraging automatic differentiation (AD) to train through these iterations, the TWNN adaptively determines the necessary computational depth for a given input. This approach enables bi-directional information flow, allowing higher-level features to refine earlier computations, effectively mitigating the vanishing gradient problem and improving efficiency.

Introduction

Traditional deep learning models, such as Residual Networks (ResNets), utilize a fixed number of layers regardless of whether an input is "easy" or "difficult" to classify. This static depth is often computationally inefficient and susceptible to the vanishing gradient problem, where signals from the loss function diminish before reaching earlier layers. While Recurrent Neural Networks (RNNs) and Memory Augmented Neural Networks (MANNs) attempt to address long-term dependencies, they frequently suffer from memory instability, where new information prematurely overwrites useful prior knowledge.

The Temporal Wormhole Neural Network (TWNN) aims to provide a flexible architecture that is deep when necessary and shallow otherwise. By mimicking the brain's afferent and efferent pathways, it allows for bi-directional information flow, enabling the network to re-evaluate earlier computations based on high-level features discovered later in the process.

The Temporal Wormhole Architecture

The core innovation of the TWNN is the Fixed-Point-Iterator block. Unlike standard feed-forward layers, this block repeatedly applies a transformation until the output "settles" (meaning the difference between consecutive iterations falls below a predefined tolerance threshold $\epsilon$) or a maximum iteration count is reached.

Architecture of the TWNN featuring Fixed-Iteration-Loop blocks and feedback connections.

A key feature of this architecture is the Temporal Wormhole. These are feedback connections that push information from later layers back to earlier ones. This creates a "shortcut" through time and depth, allowing the network to refine its initial processing based on more complex features identified downstream.

Training via AD Transformation

To train a network with dynamic iteration counts, the authors employ Algorithmic Differentiation (AD). Standard backpropagation through time can be unstable for long or variable sequences. The TWNN uses a mathematical framework to differentiate specifically through the fixed-point iteration process. This allows the model to calculate accurate gradients for the backward pass even though the number of "steps" taken in the forward pass is dynamic and input-dependent.

Computational graph for the Automatic Differentiation (AD) transformation of the fixed-point iteration.

Experimental Evaluation

The model was evaluated on image classification (MNIST/CIFAR) and time-series generation (Sine Wave) tasks. Key results include:

  • Superior Accuracy: In classification tasks, the TWNN achieved 96.54% accuracy, significantly outperforming ResNet-36 (88.25%) and ResNet-50 (90.63%).
  • Efficiency: The model demonstrates lower memory consumption and faster average training times because it can skip unnecessary processing for simpler inputs.
  • Gradient Stability: The fixed-point AD transformation effectively mitigated the vanishing gradient problem, allowing the model to learn long-term dependencies in the sine wave task within just 300 iterations.

Discussion and Future Work

While the TWNN offers significant advantages in terms of dynamic depth and bi-directional flow, it introduces new challenges. The computational overhead per iteration must be balanced against the gains from reduced layer counts. Additionally, performance is sensitive to hyperparameters like the tolerance threshold ($\epsilon$).

Future research will focus on scaling this architecture to massive datasets like ImageNet and exploring its integration into other popular models, such as Transformers or Gated Recurrent Units (GRUs), to provide them with adaptive "thinking" capabilities.

Summary updated

Abstract

This work proposes a Neural Network model that can control its depth using an iterate-to-fixed-point operator. The architecture starts with a standard layered Network but with added connections from current later to earlier layers, along with a gate to make them inactive under most circumstances. These “temporal wormhole” connections create a shortcut that allows the Neural Network to use the information available at deeper layers and re-do earlier computations with modulated inputs. End-to-end training is accomplished by using appropriate calculations for a numeric iterate-to-fixed-point operator. In a typical case, where the “wormhole” connections are inactive, this is inexpensive; but when they are active, the network takes a longer time to settle down, and the gradient calculation is also more laborious, with an effect similar to making the network deeper. In contrast to the existing skip-connection concept, this proposed technique enables information to flow up and down in the network. Furthermore, the flow of information follows a fashion that seems analogous to the afferent and efferent flow of information through layers of processing in the brain. We evaluate models that use this novel mechanism on different long-term dependency tasks. The results are competitive with other studies, showing that the proposed model con-tributes significantly to overcoming traditional deep learning models' vanishing gradient descent problem. At the same time, the training time is significantly reduced, as the “easy” input cases are processed more quickly than “difficult” ones.
BibTeX
@inproceedings{habiba2021neural,
  author = {Mansura Habiba and Barak A. Pearlmutter},
  title = {{Neural Network based on Automatic Differentiation Transformation of Numeric Iterate-to-Fixedpoint}},
  booktitle = {2021},
  year = {2021},
  doi = {10.1109/icecet52533.2021.9698707},
}
Figure