Complex Unitary Recurrent Neural Networks Using Scaled Cayley Transform

Kehelwala Dewage Gayan Maduranga, Kyle E. Helfrich, Qiang YeAAAI 2019 · DOI

Introduction

Training Recurrent Neural Networks (RNNs) on long sequences is notoriously difficult due to the vanishing and exploding gradient problem. As gradients are backpropagated through time, they tend to either diminish to zero or grow exponentially, preventing the network from learning long-term dependencies. A robust solution involves constraining the recurrent weight matrix to be orthogonal (in the real domain) or unitary (in the complex domain), which preserves the norm of the hidden state and ensures stable gradient flow.

While previous models like the scaled Cayley orthogonal RNN (scoRNN) successfully employed these constraints, they relied on a fixed scaling matrix $D$ that required manual hyperparameter tuning. This paper introduces the Scaled Cayley Unitary RNN (scuRNN), which extends the architecture into the complex domain to make the entire parameterization fully differentiable and learnable.

The scuRNN Architecture

The scuRNN parameterizes a unitary matrix $W$ using a scaled Cayley transform. According to linear algebra, any unitary matrix can be decomposed as: $$W = (I + A)^{-1}(I - A)D$$ where $A$ is a skew-Hermitian matrix ($A^H = -A$) and $D$ is a unitary diagonal scaling matrix.

In the real-valued scoRNN, $D$ was restricted to diagonal entries of $\pm 1$, making it non-differentiable. By moving to the complex domain, the diagonal entries of $D$ can be represented as $e^{i\theta_j}$. Since $\theta_j$ is a continuous real parameter, the scaling matrix becomes differentiable. This allows scuRNN to learn the optimal scaling for a given task automatically via gradient descent using Wirtinger calculus.

The modReLU Singularity

A significant contribution of this work is the mathematical analysis of the modReLU activation function, defined as: $$\text{modReLU}(z) = \text{ReLU}(|z| + b) \frac{z}{|z|}$$ The authors identify a critical instability: when the bias $b$ is positive, the gradient of modReLU with respect to $z$ contains a singularity at the origin ($z=0$).

The gradient of the modReLU activation function exhibits a sharp spike at the origin when the bias is positive, leading to numerical instability.

As shown in the figure above, the gradient magnitude explodes near $z=0$. This explains the frequent "NaN" errors encountered in previous research. To mitigate this, the authors suggest initializing hidden states away from the origin and carefully managing the bias parameters.

Experimental Evaluation

The scuRNN was evaluated against several baseline models, including LSTM and various unitary/orthogonal RNNs, across multiple benchmarks.

Pixel-by-Pixel MNIST

In the Permuted MNIST task, pixels are fed sequentially in a fixed random order, destroying local spatial correlations and requiring the model to remember long-range patterns.

Test set accuracy on the Permuted MNIST task. scuRNN (green) achieves competitive performance with state-of-the-art orthogonal models.

As illustrated in the results, scuRNN (green line) achieves a high accuracy of 96.2%, matching or exceeding the performance of models like scoRNN while benefiting from a fully learnable parameterization.

TIMIT Speech and Synthetic Tasks

On the TIMIT speech dataset, scuRNN significantly outperformed the LSTM, achieving an Evaluation MSE of 1.67 compared to the LSTM's 8.86. In synthetic benchmarks like the Copying and Adding problems, scuRNN demonstrated superior ability to bridge long temporal gaps (up to $T=2000$ steps) where standard LSTMs typically fail to beat the baseline.

Conclusion

The Scaled Cayley Unitary RNN (scuRNN) provides a powerful framework for training stable, long-memory RNNs. By leveraging complex-valued representations, it transforms the discrete scaling problem of orthogonal RNNs into a continuous optimization task. Furthermore, the theoretical analysis of modReLU provides essential guidance for avoiding numerical instabilities in complex-valued neural networks. Future work includes developing more stable activation functions and applying scuRNN to naturally complex-valued signals like MRI data.

Summary updated

BibTeX
@inproceedings{maduranga2019complex,
  author = {Kehelwala Dewage Gayan Maduranga and Kyle E. Helfrich and Qiang Ye},
  title = {{Complex Unitary Recurrent Neural Networks Using Scaled Cayley Transform}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33014528},
}
Figure