Abstract
Recurrent Neural Networks (RNNs) are essential for processing sequential data, yet they are fundamentally limited by the vanishing and exploding gradient problem. This paper introduces the AntisymmetricRNN, an architecture derived from the perspective of RNNs as discretized Ordinary Differential Equations (ODEs). By constraining the hidden-to-hidden weight matrices to be antisymmetric, the model ensures that the underlying dynamical system is stable and preserves signal magnitude over long time horizons. Our experiments on sequential image classification tasks, including permuted MNIST and noise-padded CIFAR-10, demonstrate that AntisymmetricRNNs significantly outperform LSTMs in long-term memory tasks while requiring up to seven times fewer parameters.
Introduction
The "vanishing and exploding gradient" problem has long been the primary obstacle in training RNNs for long-term dependencies. When an error signal is propagated back through many time steps, it tends to either diminish to zero or grow exponentially, making it impossible for the network to learn relationships between distant events. While architectures like Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) use gating mechanisms to facilitate information flow, they do not fundamentally address the stability of the underlying linear dynamics and are often computationally expensive.
Recent research has explored forcing weight matrices to be orthogonal to preserve signal length. However, maintaining orthogonality during training is mathematically complex and can limit the model's expressive capacity. This paper proposes a simpler, more principled approach: designing the RNN architecture to inherit the stability properties of a continuous-time dynamical system.
Methods: Stability through Antisymmetry
The authors bridge the gap between deep learning and classical physics by viewing the hidden state update of an RNN as a discretized version of an ODE. Using the forward Euler method, the update rule is expressed as: $$h_t = h_{t-1} + \epsilon \cdot \sigma(W_h h_{t-1} + W_x x_t + b)$$ where $\epsilon$ represents the discretization step size.
The Antisymmetric Constraint
The core innovation is the structure of the hidden state weight matrix $W_h$. The authors propose using an antisymmetric matrix, constructed as $A = W - W^T$. In the context of dynamical systems, a linear system $\dot{h} = Ah$ is stable if the eigenvalues of $A$ have real parts less than or equal to zero. Antisymmetric matrices have purely imaginary eigenvalues, which means the "energy" of the system is conserved—it neither grows nor decays, but instead rotates or oscillates.
Diffusion and Gating
Because the forward Euler method can be numerically unstable, the authors introduce a small diffusion term ($-\gamma I$) to the weights. This gently nudges the eigenvalues into the stable left-half plane (real part $< 0$), ensuring the network remains stable even over very long sequences.
Additionally, the authors propose a gated variant of the model. This version uses an input gate to control how much new information enters the state at each step, while the underlying antisymmetric structure maintains the stability of the stored information.
Experimental Results
The researchers evaluated the AntisymmetricRNN on several challenging benchmarks designed to test long-term memory:
- Permuted Pixel-by-Pixel MNIST (pMNIST): In this task, the model classifies a digit by looking at one pixel at a time in a shuffled order (784 steps). The gated AntisymmetricRNN achieved 98.8% accuracy, outperforming the standard LSTM (97.3%) while using only 1/7th of the parameters (10k vs 68k).
- Noise-padded CIFAR-10: This "stress test" involves processing 32 steps of image data followed by 968 steps of pure random noise. The model must "hold its breath" and remember the image through the interference. While the LSTM failed completely (11.6% accuracy), the gated AntisymmetricRNN maintained its memory, achieving 54.7% accuracy.
Efficiency and Predictability
Beyond accuracy, the AntisymmetricRNN is highly parameter-efficient. Since an antisymmetric matrix is defined by its upper triangle, it requires roughly half the parameters of a standard weight matrix. Furthermore, visualizations of hidden states revealed that while vanilla RNNs often exhibit chaotic trajectories, AntisymmetricRNN states follow smooth, predictable paths, making them significantly easier for optimizers to navigate.
Conclusion
The AntisymmetricRNN demonstrates that a principled approach based on dynamical systems can solve long-standing problems in recurrent neural networks. By ensuring stability by design, the model achieves state-of-the-art performance on long-term memory tasks with minimal parameter overhead. Future research could explore more sophisticated numerical solvers, such as Runge-Kutta methods, to further improve the discretization of the underlying ODE.
Summary updated