A Differential Geometry Perspective on Orthogonal Recurrent Models

Omri Azencot, N. Benjamin Erichson, Mirela Ben-Chen, Michael W. MahoneyarXiv 2021 · arXiv

Introduction

Orthogonal Recurrent Neural Networks (RNNs) have become a standard solution for mitigating the vanishing and exploding gradient problems in sequence modeling. By constraining hidden-to-hidden weight matrices to be orthogonal, these models preserve the norm of the hidden state across time steps. However, this stability often comes at the expense of expressivity, as the model is restricted to a specific mathematical manifold. This paper provides a novel differential geometry perspective, revealing that orthogonal RNNs are mathematically equivalent to optimizing within the space of divergence-free vector fields. Building on this insight, the authors introduce the Latent Vector Field RNN (lvfRNN), which generalizes these constraints to span a broader space of vector fields while maintaining training stability.

The Geometric Bridge

The core contribution of this work is the link between deep learning and dynamical systems. The authors prove that a vector field is divergence-free—meaning it lacks "sources" or "sinks" that cause information to accumulate or dissipate—if and only if its corresponding operator is skew-symmetric. Since skew-symmetric matrices generate orthogonal matrices, orthogonal RNNs can be viewed as simulations of incompressible flows.

The action of a vector field v on a latent function f, illustrating the transport equation framework.

The lvfRNN parameterizes the hidden state transition using a transport equation: $\partial_t f = -\mathcal{D}_{\mathbf{v}} f$. This formulation allows the model to decompose the transition into a rotation component (skew-symmetric) and a flux component (diagonal), enabling it to both preserve and process information dynamically.

Discrete Latent Spaces and Stability

To implement this continuous framework in a neural network, the authors develop a discrete geometry toolbox. They define the latent space as a graph where nodes represent states, and edges represent transitions. This allows for the definition of discrete versions of the gradient, divergence, and directional derivatives.

A discrete representation of the latent space as a graph, where operators are defined over nodes and edges.

A critical finding is that strict orthogonality is not strictly necessary for stability. Instead, the model remains stable as long as the divergence of the underlying vector field is non-positive. This "soft" constraint provides significantly more flexibility than the rigid Stiefel manifold constraints used in traditional orthogonal RNNs.

Empirical Performance

The lvfRNN was evaluated against state-of-the-art orthogonal models (such as expRNN and nnRNN) across several benchmarks:

  • Language Modeling (PTB): Achieved a state-of-the-art 1.43 Bits Per Character (BPC), outperforming the previous best orthogonal model (1.47 BPC).
  • Polyphonic Music (JSB & MuseData): Significantly outperformed competitors in negative log-likelihood, demonstrating a superior ability to capture complex non-linear dynamics.
  • Speech Prediction (TIMIT): Performed comparably to specialized models like expRNN.
  • Memory Tasks (Copy Task): Successfully handled long-term dependencies, achieving ~95% accuracy compared to the ~16% of standard RNNs.

Conclusion

By reframing orthogonality through the lens of physics and differential geometry, this work provides a deeper intuition for why certain RNN architectures succeed. The lvfRNN demonstrates that controlling divergence, rather than enforcing strict orthogonality, is the key to balancing stability and expressivity. While current implementations use simple Euler integration, this framework opens the door for applying more sophisticated tools from PDE theory and differential geometry to neural architecture design.

Summary updated

Abstract

Recently, orthogonal recurrent neural networks (RNNs) have emerged as state-of-the-art models for learning long-term dependencies. This class of models mitigates the exploding and vanishing gradients problem by design. In this work, we employ tools and insights from differential geometry to offer a novel perspective on orthogonal RNNs. We show that orthogonal RNNs may be viewed as optimizing in the space of divergence-free vector fields. Specifically, based on a well-known result in differential geometry that relates vector fields and linear operators, we prove that every divergence-free vector field is related to a skew-symmetric matrix. Motivated by this observation, we study a new recurrent model, which spans the entire space of vector fields. Our method parameterizes vector fields via the directional derivatives of scalar functions. This requires the construction of latent inner product, gradient, and divergence operators. In comparison to state-of-the-art orthogonal RNNs, our approach achieves comparable or better results on a variety of benchmark tasks.
BibTeX
@misc{azencot2021differential,
  author = {Omri Azencot and N. Benjamin Erichson and Mirela Ben-Chen and Michael W. Mahoney},
  title = {{A Differential Geometry Perspective on Orthogonal Recurrent Models}},
  year = {2021},
  eprint = {2102.09589},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.09589},
}
Figure