Introduction
Orthogonal Recurrent Neural Networks (RNNs) have become a standard solution for mitigating the vanishing and exploding gradient problems in sequence modeling. By constraining hidden-to-hidden weight matrices to be orthogonal, these models preserve the norm of the hidden state across time steps. However, this stability often comes at the expense of expressivity, as the model is restricted to a specific mathematical manifold. This paper provides a novel differential geometry perspective, revealing that orthogonal RNNs are mathematically equivalent to optimizing within the space of divergence-free vector fields. Building on this insight, the authors introduce the Latent Vector Field RNN (lvfRNN), which generalizes these constraints to span a broader space of vector fields while maintaining training stability.
The Geometric Bridge
The core contribution of this work is the link between deep learning and dynamical systems. The authors prove that a vector field is divergence-free—meaning it lacks "sources" or "sinks" that cause information to accumulate or dissipate—if and only if its corresponding operator is skew-symmetric. Since skew-symmetric matrices generate orthogonal matrices, orthogonal RNNs can be viewed as simulations of incompressible flows.
The lvfRNN parameterizes the hidden state transition using a transport equation: $\partial_t f = -\mathcal{D}_{\mathbf{v}} f$. This formulation allows the model to decompose the transition into a rotation component (skew-symmetric) and a flux component (diagonal), enabling it to both preserve and process information dynamically.
Discrete Latent Spaces and Stability
To implement this continuous framework in a neural network, the authors develop a discrete geometry toolbox. They define the latent space as a graph where nodes represent states, and edges represent transitions. This allows for the definition of discrete versions of the gradient, divergence, and directional derivatives.
A critical finding is that strict orthogonality is not strictly necessary for stability. Instead, the model remains stable as long as the divergence of the underlying vector field is non-positive. This "soft" constraint provides significantly more flexibility than the rigid Stiefel manifold constraints used in traditional orthogonal RNNs.
Empirical Performance
The lvfRNN was evaluated against state-of-the-art orthogonal models (such as expRNN and nnRNN) across several benchmarks:
- Language Modeling (PTB): Achieved a state-of-the-art 1.43 Bits Per Character (BPC), outperforming the previous best orthogonal model (1.47 BPC).
- Polyphonic Music (JSB & MuseData): Significantly outperformed competitors in negative log-likelihood, demonstrating a superior ability to capture complex non-linear dynamics.
- Speech Prediction (TIMIT): Performed comparably to specialized models like expRNN.
- Memory Tasks (Copy Task): Successfully handled long-term dependencies, achieving ~95% accuracy compared to the ~16% of standard RNNs.
Conclusion
By reframing orthogonality through the lens of physics and differential geometry, this work provides a deeper intuition for why certain RNN architectures succeed. The lvfRNN demonstrates that controlling divergence, rather than enforcing strict orthogonality, is the key to balancing stability and expressivity. While current implementations use simple Euler integration, this framework opens the door for applying more sophisticated tools from PDE theory and differential geometry to neural architecture design.
Summary updated