Introduction
Training Recurrent Neural Networks (RNNs) on long sequences is notoriously difficult due to the vanishing and exploding gradient problem. As gradients are backpropagated through time, they tend to either diminish to zero or grow exponentially, preventing the network from learning long-term dependencies. A robust solution involves constraining the recurrent weight matrix to be orthogonal (in the real domain) or unitary (in the complex domain), which preserves the norm of the hidden state and ensures stable gradient flow.
While previous models like the scaled Cayley orthogonal RNN (scoRNN) successfully employed these constraints, they relied on a fixed scaling matrix $D$ that required manual hyperparameter tuning. This paper introduces the Scaled Cayley Unitary RNN (scuRNN), which extends the architecture into the complex domain to make the entire parameterization fully differentiable and learnable.
The scuRNN Architecture
The scuRNN parameterizes a unitary matrix $W$ using a scaled Cayley transform. According to linear algebra, any unitary matrix can be decomposed as: $$W = (I + A)^{-1}(I - A)D$$ where $A$ is a skew-Hermitian matrix ($A^H = -A$) and $D$ is a unitary diagonal scaling matrix.
In the real-valued scoRNN, $D$ was restricted to diagonal entries of $\pm 1$, making it non-differentiable. By moving to the complex domain, the diagonal entries of $D$ can be represented as $e^{i\theta_j}$. Since $\theta_j$ is a continuous real parameter, the scaling matrix becomes differentiable. This allows scuRNN to learn the optimal scaling for a given task automatically via gradient descent using Wirtinger calculus.
The modReLU Singularity
A significant contribution of this work is the mathematical analysis of the modReLU activation function, defined as: $$\text{modReLU}(z) = \text{ReLU}(|z| + b) \frac{z}{|z|}$$ The authors identify a critical instability: when the bias $b$ is positive, the gradient of modReLU with respect to $z$ contains a singularity at the origin ($z=0$).
As shown in the figure above, the gradient magnitude explodes near $z=0$. This explains the frequent "NaN" errors encountered in previous research. To mitigate this, the authors suggest initializing hidden states away from the origin and carefully managing the bias parameters.
Experimental Evaluation
The scuRNN was evaluated against several baseline models, including LSTM and various unitary/orthogonal RNNs, across multiple benchmarks.
Pixel-by-Pixel MNIST
In the Permuted MNIST task, pixels are fed sequentially in a fixed random order, destroying local spatial correlations and requiring the model to remember long-range patterns.
As illustrated in the results, scuRNN (green line) achieves a high accuracy of 96.2%, matching or exceeding the performance of models like scoRNN while benefiting from a fully learnable parameterization.
TIMIT Speech and Synthetic Tasks
On the TIMIT speech dataset, scuRNN significantly outperformed the LSTM, achieving an Evaluation MSE of 1.67 compared to the LSTM's 8.86. In synthetic benchmarks like the Copying and Adding problems, scuRNN demonstrated superior ability to bridge long temporal gaps (up to $T=2000$ steps) where standard LSTMs typically fail to beat the baseline.
Conclusion
The Scaled Cayley Unitary RNN (scuRNN) provides a powerful framework for training stable, long-memory RNNs. By leveraging complex-valued representations, it transforms the discrete scaling problem of orthogonal RNNs into a continuous optimization task. Furthermore, the theoretical analysis of modReLU provides essential guidance for avoiding numerical instabilities in complex-valued neural networks. Future work includes developing more stable activation functions and applying scuRNN to naturally complex-valued signals like MRI data.
Summary updated