Complex Evolution Recurrent Neural Networks (ceRNNs)

Izhak Shafran, Tom Bagby, R. J. Skerry-Ryan

TL;DR

This paper investigates whether the strict "unitary" constraint in Unitary Evolution Recurrent Neural Networks (uRNNs) is actually necessary for their success. By introducing Complex Evolution RNNs (ceRNNs)—which retain complex-valued math but drop the unitary constraint—the authors show that flexibility in information decay actually improves performance on real-world tasks like large-scale speech recognition.

What Problem Does This Paper Address?

Standard Recurrent Neural Networks (RNNs) often suffer from exploding or vanishing gradients, making it difficult for them to learn long-term dependencies. To solve this, researchers previously proposed uRNNs, which constrain the transition matrices to be unitary (preserving the norm of the hidden state). While uRNNs excel at synthetic memory benchmarks, they haven't been widely tested on massive, real-world datasets.

The authors of this paper ask a fundamental question: Is the "unitary" property a feature or a bug? While it prevents gradients from vanishing, it also prevents the network from "forgetting" information. In tasks like Automatic Speech Recognition (ASR), where the model needs to focus on recent acoustic signals and ignore distant noise, the inability to decay information might actually be a disadvantage.

Methods

The authors propose ceRNNs, a middle ground between standard RNNs and uRNNs.

  1. The Cascade Approach: Like uRNNs, ceRNNs represent the weight matrix $W$ as a "cascade" of efficient linear operators: Fourier transforms ($\mathcal{F}$), permutations ($P$), and Householder reflections ($R$). This keeps the parameter count low—growing linearly $O(N)$ rather than quadratically $O(N^2)$.
  2. Dropping the Constraint: In a uRNN, the diagonal matrices ($D$) in this cascade are strictly constrained to lie on the unit circle in the complex plane (magnitude = 1). In ceRNNs, this constraint is removed. The diagonal elements can take any complex value, allowing the model to learn to amplify or attenuate specific components of the hidden state.
  3. Complex Activation: Both models use modReLu, a complex-valued extension of the standard ReLU that applies a bias to the magnitude of the complex number while preserving its phase.

Datasets

The researchers evaluated the models across three distinct scales:

  • Multivariate Linear Regression: A synthetic task used to visualize learning trajectories and convergence stability.
  • Copy Memory Task: A classic benchmark for testing a model's ability to remember a sequence over a long "filler" period.
  • Large-Scale ASR: 15,000 hours of anonymized Google Voice Search queries. This includes a "clean" test set and a "noisy" set (corrupted with additive noise and reverberation).

Key Insights and Ideas

  • Unitarity is not the "Secret Sauce" for Memory: Surprisingly, ceRNNs and uRNNs performed identically on the Copy Memory task. This suggests that the superior performance of these models over LSTMs is due to their complex-valued nature and the specific structure of their linear operators, rather than the strict unitary constraint itself.
  • Strict Unitarity Can Hurt Real-World Performance: In the ASR task, adding a uRNN layer actually degraded the performance of a baseline LSTM. The authors hypothesize that the unitary constraint forces the model to retain too much "history," making it harder to process the current speech signal.
  • Stability through Flexibility: During training for linear regression, uRNNs showed erratic "spikes" in loss. These spikes coincided with the model's parameters making abrupt transitions to satisfy the unitary constraint. ceRNNs, being unconstrained, showed much smoother convergence.

Key Results

  • Speech Recognition (ASR): Pre-pending a ceRNN layer to a baseline LSTM improved the Word Error Rate (WER) by 0.8% absolute on the noisy test set. In contrast, the uRNN version increased the error rate by nearly 1% compared to the baseline.
  • Efficiency: Because ceRNNs use an $O(N)$ parameterization, the authors were able to scale the hidden layer size from 512 to 2048 nodes with only a marginal increase in total parameters (adding only ~14k parameters), leading to further WER gains.
  • Convergence: In the Copy Memory task, both ceRNNs and uRNNs reached a cross-entropy loss of 0.18, significantly outperforming the standard LSTM's 0.46.

Limitations and Future Work

The authors note that while ceRNNs provide a boost, they did not explore the optimal "recipe" for the operator cascade. The specific sequence of Fourier transforms and reflections used was inherited from prior uRNN work without much justification.

Future research could investigate:

  • Architectural Search: Finding more effective combinations of complex-valued operators.
  • Signal Processing Integration: Since ceRNNs utilize complex-valued FFTs, there is a promising opportunity to integrate them directly with front-end signal processing tasks like de-reverberation and beam-forming within a single end-to-end framework.

Summary updated

Abstract

Unitary Evolution Recurrent Neural Networks (uRNNs) have three attractive properties: (a) the unitary property, (b) the complex-valued nature, and (c) their efficient linear operators [1]. The literature so far does not address - how critical is the unitary property of the model? Furthermore, uRNNs have not been evaluated on large tasks. To study these shortcomings, we propose the complex evolution Recurrent Neural Networks (ceRNNs), which is similar to uRNNs but drops the unitary property selectively. On a simple multivariate linear regression task, we illustrate that dropping the constraints improves the learning trajectory. In copy memory task, ceRNNs and uRNNs perform identically, demonstrating that their superior performance over LSTMs is due to complex-valued nature and their linear operators. In a large scale real-world speech recognition, we find that pre-pending a uRNN degrades the performance of our baseline LSTM acoustic models, while pre-pending a ceRNN improves the performance over the baseline by 0.8% absolute WER.
BibTeX
@inproceedings{shafran2018complex,
  author = {Izhak Shafran and Tom Bagby and R. J. Skerry-Ryan},
  title = {{Complex Evolution Recurrent Neural Networks (ceRNNs)}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=lfgw1BGpxv0},
}

Loading related content...

Figure