Time Adaptive Recurrent Neural Network

Anil Kag, Venkatesh Saligrama

Abstract

We propose a learning method that dynamically modifies the time-constants of the continuous-time counterpart of a vanilla RNN. The time-constants are modified based on the current observation and hidden state. Our proposal overcomes the issues of RNN trainability by mitigating exploding and vanishing gradient phenomena based on placing novel constraints on the parameter space, and by suppressing noise in inputs based on pondering over informative inputs to strengthen their contribution in the hidden state. As a result, our method is computationally efficient, overcoming overheads of many existing methods that also attempt to improve RNN training. Our RNNs, despite being simpler and having a light memory footprint, show competitive performance against standard LSTMs and baseline RNN models on many benchmark datasets, including those that require long-term memory.

Introduction

Vanilla Recurrent Neural Networks (RNNs) are highly desirable for resource-constrained environments, such as IoT edge devices, due to their minimal memory requirements. However, they are plagued by two fundamental challenges. First, the gradient problem: during backpropagation through time, gradients often vanish or explode, hindering the learning of long-term dependencies (LTD). Second, the noise problem: real-world sequential data is frequently "bursty," containing sparse informative signals buried within long sequences of irrelevant noise. Standard RNNs process every timestep with equal weight, which often causes critical information to be "washed out" by noise.

While architectures like LSTMs address these issues through complex gating mechanisms, they introduce significant computational overhead. The Time Adaptive Recurrent Neural Network (TARNN) addresses these flaws by treating RNN state updates as an Ordinary Differential Equation (ODE) with a learnable, time-varying time-constant.

Methodology

The TARNN is formulated as a continuous-time RNN modeled by an ODE. The core innovation is the introduction of a time-adaptive time-constant ($\lambda$).

ODE Formulation and Dynamic Pondering

Instead of discrete updates, the hidden state $z(t)$ evolves according to: $$\lambda(u_m) \circ \dot{z}(t) = Az(t) + Bu_m + \phi(Uz(t) + Wu_m)$$ where $u_m$ is a function of the current input and previous state. The parameter $\lambda$ is a learned function of the input, allowing the model to dynamically adjust its "pondering" time:

  • Large $\lambda$: The system evolves slowly, effectively "skipping" the input to preserve the previous state (ideal for noise suppression).
  • Small $\lambda$: The system reaches equilibrium quickly, "pondering" the input deeply to update the hidden state.

Lossless Information Propagation

The authors mathematically ensure gradient stability by placing specific constraints on the parameter space and employing a novel regularizer. This ensures that partial gradients of hidden states maintain a unit norm, effectively eliminating the vanishing gradient problem. For practical implementation, the ODE is discretized using a computationally efficient 3-step Euler recursion.

Experiments and Results

The TARNN was evaluated on diverse benchmarks, including synthetic LTD tasks (Add & Copy), sequential image classification (Pixel/Permute-MNIST), noisy benchmarks (Noisy-MNIST/CIFAR), and language modeling (PTB).

Figure 2: Performance on a synthetic long-term dependency task. (a) TARNN state updates align with informative inputs. (b) Gradient stability during training. (c) Accuracy comparison showing TARNN's 100% success rate.

Performance on Long-Term Dependencies

On a synthetic task with sequence length 16 where only two timesteps were informative, TARNN achieved 100% accuracy. In contrast, LSTMs (45%) and other specialized RNNs (37-60%) struggled to isolate the relevant signals from noise. This "temporal attention" allows TARNN to focus on critical data points without the computational cost of standard attention mechanisms.

Efficiency and Convergence

TARNN demonstrates superior training efficiency. On long-term dependency tasks, it converges up to 8x faster than LSTMs. On the Noisy-CIFAR dataset, TARNN achieved 59.06% accuracy, significantly outperforming the iRNN baseline.

Figure 5: Convergence comparison on a long-sequence task (length 500). TARNN achieves lower cross-entropy significantly faster than standard LSTM and other RNN variants.

In language modeling (PTB), a TARNN with 128 hidden units outperformed an LSTM with 256 units, demonstrating higher parameter efficiency. Its vanilla-style architecture ensures a tiny memory footprint, making it ideal for edge AI applications.

Conclusion

The Time Adaptive Recurrent Neural Network offers a robust solution for processing noisy, long-term sequential data with minimal computational cost. By framing state updates as an adaptive ODE, it achieves a form of "hard" temporal attention and ensures gradient stability. While the current implementation uses a fixed 3-step Euler discretization, future work could explore more complex ODE solvers or the integration of TARNN layers into larger transformer-based architectures.

Summary updated

Abstract

We propose a learning method that, dynamically modifies the time-constants of the continuous-time counterpart of a vanilla RNN. The time-constants are modified based on the current observation and hidden state. Our proposal overcomes the issues of RNN trainability, by mitigating exploding and vanishing gradient phenomena based on placing novel constraints on the parameter space, and by suppressing noise in inputs based on pondering over informative inputs to strengthen their contribution in the hidden state. As a result, our method is computationally efficient overcoming overheads of many existing methods that also attempt to improve RNN training. Our RNNs, despite being simpler and having light memory footprint, shows competitive performance against standard LSTMs and baseline RNN models on many benchmark datasets including those that require long-term memory.
BibTeX
@inproceedings{kag2021time,
  author = {Anil Kag and Venkatesh Saligrama},
  title = {{Time Adaptive Recurrent Neural Network}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=oBk66ctzzR},
}
Figure