Abstract
This paper introduces the Temporal Wormhole Neural Network (TWNN), a novel architecture that replaces fixed-depth layers with a dynamic iterate-to-fixed-point mechanism. By leveraging automatic differentiation (AD) to train through these iterations, the TWNN adaptively determines the necessary computational depth for a given input. This approach enables bi-directional information flow, allowing higher-level features to refine earlier computations, effectively mitigating the vanishing gradient problem and improving efficiency.
Introduction
Traditional deep learning models, such as Residual Networks (ResNets), utilize a fixed number of layers regardless of whether an input is "easy" or "difficult" to classify. This static depth is often computationally inefficient and susceptible to the vanishing gradient problem, where signals from the loss function diminish before reaching earlier layers. While Recurrent Neural Networks (RNNs) and Memory Augmented Neural Networks (MANNs) attempt to address long-term dependencies, they frequently suffer from memory instability, where new information prematurely overwrites useful prior knowledge.
The Temporal Wormhole Neural Network (TWNN) aims to provide a flexible architecture that is deep when necessary and shallow otherwise. By mimicking the brain's afferent and efferent pathways, it allows for bi-directional information flow, enabling the network to re-evaluate earlier computations based on high-level features discovered later in the process.
The Temporal Wormhole Architecture
The core innovation of the TWNN is the Fixed-Point-Iterator block. Unlike standard feed-forward layers, this block repeatedly applies a transformation until the output "settles" (meaning the difference between consecutive iterations falls below a predefined tolerance threshold $\epsilon$) or a maximum iteration count is reached.
A key feature of this architecture is the Temporal Wormhole. These are feedback connections that push information from later layers back to earlier ones. This creates a "shortcut" through time and depth, allowing the network to refine its initial processing based on more complex features identified downstream.
Training via AD Transformation
To train a network with dynamic iteration counts, the authors employ Algorithmic Differentiation (AD). Standard backpropagation through time can be unstable for long or variable sequences. The TWNN uses a mathematical framework to differentiate specifically through the fixed-point iteration process. This allows the model to calculate accurate gradients for the backward pass even though the number of "steps" taken in the forward pass is dynamic and input-dependent.
Experimental Evaluation
The model was evaluated on image classification (MNIST/CIFAR) and time-series generation (Sine Wave) tasks. Key results include:
- Superior Accuracy: In classification tasks, the TWNN achieved 96.54% accuracy, significantly outperforming ResNet-36 (88.25%) and ResNet-50 (90.63%).
- Efficiency: The model demonstrates lower memory consumption and faster average training times because it can skip unnecessary processing for simpler inputs.
- Gradient Stability: The fixed-point AD transformation effectively mitigated the vanishing gradient problem, allowing the model to learn long-term dependencies in the sine wave task within just 300 iterations.
Discussion and Future Work
While the TWNN offers significant advantages in terms of dynamic depth and bi-directional flow, it introduces new challenges. The computational overhead per iteration must be balanced against the gains from reduced layer counts. Additionally, performance is sensitive to hyperparameters like the tolerance threshold ($\epsilon$).
Future research will focus on scaling this architecture to massive datasets like ImageNet and exploring its integration into other popular models, such as Transformers or Gated Recurrent Units (GRUs), to provide them with adaptive "thinking" capabilities.
Summary updated