Abstract
Unitary Evolution Recurrent Neural Networks (uRNNs) have three attractive properties: (a) the unitary property, (b) the complex-valued nature, and (c) their efficient linear operators. The literature so far does not address -- how critical is the unitary property of the model? Furthermore, uRNNs have not been evaluated on large tasks. To study these shortcomings, we propose the complex evolution Recurrent Neural Networks (ceRNNs), which is similar to uRNNs but drops the unitary property selectively. On a simple multivariate linear regression task, we illustrate that dropping the constraints improves the learning trajectory. In copy memory task, ceRNNs and uRNNs perform identically, demonstrating that their superior performance over LSTMs is due to complex-valued nature and their linear operators. In a large scale real-world speech recognition, we find that pre-pending a uRNN degrades the performance of our baseline LSTM acoustic models, while pre-pending a ceRNN improves the performance over the baseline by 0.8% absolute WER.
Introduction
Standard Recurrent Neural Networks (RNNs) often struggle with long-term dependencies due to the vanishing or exploding gradient problem. Unitary Evolution RNNs (uRNNs) were introduced to mitigate this by constraining transition matrices to be unitary, ensuring norm preservation of the hidden state. While effective on synthetic benchmarks, the necessity of the strict unitary constraint remains a question, especially for real-world tasks where information decay (forgetting) might be beneficial. This paper introduces Complex Evolution RNNs (ceRNNs), which retain the complex-valued architecture and efficient parameterization of uRNNs but relax the unitary constraint.
Methodology
The ceRNN architecture is built upon the efficient linear operator cascade used in uRNNs, comprising Fourier transforms ($\mathcal{F}$), permutations ($P$), and Householder reflections ($R$).
- Parameterization: The weight matrix $W$ is represented as a product of these operators, maintaining $O(N)$ parameter complexity.
- Relaxed Constraints: Unlike uRNNs, where diagonal matrices $D$ are constrained to the unit circle, ceRNNs allow these diagonal elements to take any complex value. This flexibility enables the network to learn to selectively amplify or attenuate components of the hidden state.
- Activation Function: Both models employ modReLu, a complex-valued activation that biases the magnitude while preserving the phase of the complex hidden state.
Experimental Evaluation
The models were tested across three scales: multivariate linear regression for stability, the Copy Memory task for long-term dependencies, and large-scale Automatic Speech Recognition (ASR) using 15,000 hours of Google Voice Search data.
Insights from Memory Benchmarks
A key finding is that the unitary property is not the primary driver of performance in memory-intensive tasks. In the Copy Memory task, ceRNNs and uRNNs achieved nearly identical performance, both significantly outperforming standard LSTMs. This indicates that the complex-valued representation and the specific structure of the linear operators are the primary factors for handling long-term dependencies.
Stability and Flexibility
Training dynamics revealed that strict unitarity can lead to instability. In linear regression tasks, uRNNs exhibited loss spikes as parameters were forced to satisfy unitary constraints. ceRNNs, by contrast, showed smoother convergence. In real-world ASR, the inability of uRNNs to "forget" distant noise proved detrimental, whereas ceRNNs' ability to decay information led to performance gains.
Key Results
The most significant results were observed in large-scale ASR. Pre-pending a ceRNN layer to a baseline LSTM acoustic model resulted in an absolute reduction of 0.8% in Word Error Rate (WER) on noisy datasets. Conversely, using a uRNN layer in the same configuration degraded performance by approximately 1% relative to the baseline.
Furthermore, the $O(N)$ parameterization of ceRNNs allowed for scaling hidden layer sizes (e.g., from 512 to 2048) with minimal parameter overhead, facilitating further accuracy improvements.
Conclusion
The investigation into ceRNNs demonstrates that the strict unitary constraint in uRNNs is often unnecessary and can be counterproductive in real-world applications like speech recognition. By relaxing this constraint while preserving complex-valued operations and efficient parameterization, ceRNNs offer a more flexible and performant alternative for sequence modeling. Future work will explore optimized operator cascades and deeper integration with front-end signal processing.
Summary updated