Stability-Paced Curricula for Tensor-Power Recurrent Networks

Recurrent networks that multiply their hidden state by itself several times at each step can remember much further into the past than ordinary recurrent networks. The catch is that the same multiplication makes training explode. Qiu et al. show that raising the "degree" of this self-multiplication above one amplifies memory, but it also makes the step-to-step derivatives unbounded (On the Memory Mechanism of Tensor-Power Recurrent Models). Their fix is to make the degree a continuous parameter that gradient descent learns. Nothing in that recipe stops the optimizer from pushing into the unstable region before the network has learned anything, or from retreating to degree one and giving up the memory.

We propose to handle the phase boundary with a curriculum, meaning a training schedule that gives the model easy material before hard material. Here the schedule is driven by the network's own measured stability rather than a hand-picked difficulty score. The permitted degree and the training sequence length rise together, and they rise only while a running estimate of how fast perturbations grow through the recurrence stays below a threshold. They fall back when that estimate crosses the threshold. We treat the training order as part of the data, so every result ships with a documented ordering log. We also require any curriculum gain to exceed the variation caused by undocumented preprocessing choices. That last requirement turns dataset transparency from an afterthought into the bar a claim has to clear.

Core Claim

The claim is that a curriculum paced by measured stability lets tensor-power recurrent models reliably reach and stay in the high-degree, long-memory regime. The same models diverge or collapse to low degree when the degree is learned freely under uniform training. A fixed schedule with the same endpoints should not achieve this reliably. The benefit should grow with how much long-range memory the task demands and disappear on short-memory tasks. This is a falsifiable, dose-dependent prediction, not a blanket "curricula help" statement.

Three things are new relative to the closest work. First, On the Memory Mechanism of Tensor-Power Recurrent Models learns the degree by gradient descent under uniform sampling; we cap the degree by a stability-gated schedule and couple it to sequence length. Second, self-paced learning (Self-Paced Learning for Latent Variable Models) admits examples according to their current loss; we reuse its idea of an adaptive admission threshold but gate on a property of the model's dynamics rather than per-example loss. Third, the tensor-recurrent papers in the file report no dataset documentation. We make the curriculum schedule and preprocessing a documented, audited artifact and use it to bound how large a gain has to be before it counts.

Research Program

The model is the differentiable tensor-power recurrence of Qiu et al. The degree-$p$ outer power of the hidden state is rewritten, through symmetric tensor decomposition, as a sum of $R$ powered inner products using the sign-preserving power $\phi_p(s)=\mathrm{sgn}(s)|s|^p$, so $p$ is a real number.

The stability signal is a finite-time growth rate. It is estimated on a small probe batch every few hundred steps:

$$\gamma = \frac{1}{T}\log \Big\| \prod_{t=1}^{T} \frac{\partial h_t}{\partial h_{t-1}} \Big\|,$$

where $h_t$ is the hidden state at step $t$, $T$ is the current training sequence length, and the norm is the spectral norm estimated by a few power iterations with vector-Jacobian products. A positive $\gamma$ means perturbations grow exponentially along the sequence. This is exactly the failure mode Qiu et al. identify for $p>1$.

The curriculum keeps a stage index $k$. Each stage sets a degree cap $p_{\max}(k)$ and a sequence length $T(k)$, both increasing in $k$. The learnable $p$ is clamped to $[1, p_{\max}(k)]$. The rule for moving between stages is simple. The model advances one stage when $\gamma < \tau$ has held for $K$ consecutive checks, and it retreats one stage and restores the last stable checkpoint's optimizer state when $\gamma > \tau_{\text{high}}$. The thresholds $\tau$, $\tau_{\text{high}}$ and patience $K$ are the only new hyperparameters. We will report sensitivity to all three, because pacing sensitivity is a known weakness of curricula.

We will also analyze where runs end up. For every run we record the trajectory of $(p, \gamma, T)$. We then ask whether uniform-training runs that fail do so by blowing up (γ spikes, loss diverges) or by retreating to $p\approx 1$. The second failure is the recurrent analogue of the low-capacity "simple pockets" that Generalized Tensor Models for Recurrent Neural Networks found in deep ReLU recurrent networks. That paper motivates the question but does not propose a remedy. We will also check whether the curriculum changes the "rank paradox" Qiu et al. report, in which more decomposition branches $R$ flatten the loss landscape without adding memory. It is possible that the paradox is partly an optimization artifact that disappears once the high-$p$ regime is reachable.

The transparency component has two parts. The first is a written datasheet addendum for each time-series benchmark, following the lifecycle questions of Datasheets for datasets. It records source, windowing, normalization, missing-value handling and train/test splitting. Its "preprocessing" and "uses" sections are extended with the training-order schedule, which Gebru et al.'s static-dataset template does not cover. Gebru et al. stress deliberate human reflection over automation. We therefore keep the datasheet manual and add, separately, a machine-readable log of the realized stage trajectory, so a replicator can reproduce the exact data order. The second part is a preprocessing sensitivity study, motivated by Reproducibility of deep learning in digital pathology whole slide image analysis, where undocumented preprocessing alone broke replication. We vary plausible but typically unreported choices, namely window stride, normalization scope and train/test boundary, and measure the spread in test error they cause. A curriculum gain smaller than that spread is treated as not established.

Baselines

The naive reference is a standard LSTM together with the tensor-power model at fixed degree $p=1$, trained with uniform sampling at full sequence length. This shows what the task yields with no memory amplification.

The strongest existing method is the differentiable tensor-power model of On the Memory Mechanism of Tensor-Power Recurrent Models, with $p$ learned freely and uniform sampling. It is the direct competitor because it already addresses the same memory-stability trade-off and reports the lowest error rates against vanilla recurrent networks, LSTMs and multiplicative recurrent networks on ARFIMA, tree-ring and traffic data. It is also reportedly about ten times faster than the multiplicative-recurrent baseline. No reported numeric RMSE or MAE for it was found on Lacuna, so we will re-run it ourselves. It gets the same data splits, rank $R$, optimizer and an equal hyperparameter-tuning budget, measured in trials. Its tuning space includes the initial value of $p$ and gradient clipping, so that it is not handicapped. Beating it means lower median test error across at least ten seeds and a lower fraction of diverged runs, with the error gap exceeding the preprocessing-induced spread defined above.

Several ablations isolate the key ingredient, which is stability gating. A fixed linear schedule uses the same final $p_{\max}$ and $T$ and the same total steps but ignores $\gamma$. A length-only curriculum keeps $p$ free, and a degree-only curriculum keeps $T$ at full length. A stability-regularized variant penalizes $\gamma$ in the loss instead of gating the schedule. All variants share the per-run compute budget, and the cost of the $\gamma$ probes is charged to the curriculum methods.

A secondary transfer baseline comes from video. On UCF11, TT-GRU reports 81.3% accuracy (Tensor-Train Recurrent Neural Networks for Video Classification) and BT-LSTM reports 85.3% (Learning Compact Recurrent Neural Networks With Block-Term Tensor Decomposition). We will test only the length half of the curriculum there, since those models have no degree parameter. A result counts only if it improves a re-run under our own documented preprocessing, rather than the published figure.

Why not just…?

Why not just initialize $p$ at 1 and let gradient descent raise it? This is the most serious threat. Gradient descent starting from a stable point may already act as an implicit curriculum. The free-$p$ baseline has initialization in its tuning space for exactly this reason. If it matches our method on divergence rate and on the learned $p$ at high memory demand, the proposal fails. The trajectory analysis will show whether free-$p$ runs stall near $p\approx 1$, which would favor our method, or climb smoothly, which would support the alternative.

Why not just clip gradients or penalize the growth rate? Clipping bounds the update size but not the forward dynamics. A network whose hidden state blows up along the sequence can still be clipped into a useless region. Penalizing $\gamma$ is a real alternative and is included as an ablation. If it matches gating, the contribution shrinks to "measure $\gamma$", which is still useful but a smaller claim, and we would report it that way.

Why not just constrain the recurrence to be orthogonal, as in Orthogonal Tensor Trains? Scaling Recurrent Models via Orthogonal Approximations in Tensor Trains keeps tensor-train cores orthogonal through Cayley transforms and gets faster, more confident training on longitudinal MRI. That constraint stabilizes linear transitions. The instability in tensor-power models, however, comes from the power nonlinearity, which orthogonality does not bound. We will add an orthogonal-core variant of the free-$p$ baseline to test this directly.

Why not just use a plain sequence-length curriculum? Easy-to-hard ordering by length is the obvious default, and theory supports easy-to-hard progression in other settings (A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula). The length-only ablation measures what coupling degree to length adds. The broader worry, that curricula only speed up early training in large models, is less relevant here because tensor-power recurrent models are small. We nonetheless report final error, not just speed.

Decisive Checks

The central experiment uses synthetic ARFIMA series, in which the fractional differencing parameter $d$ sets how slowly correlations decay and so directly sets the memory demand. We sweep $d$ from near zero to near 0.5. The prediction is that the curriculum's advantage over free-$p$ training is near zero at small $d$ and grows with $d$. We also predict that the learned final $p$ under the curriculum rises with $d$, while under free training it saturates or diverges. A flat or absent dose-response falsifies the core claim.

We then move to the real long-memory series used by Qiu et al., namely TSDL tree rings and UCI traffic. We report RMSE, divergence fraction and seed variance, each accompanied by its datasheet and preprocessing-sensitivity band. Finally, the UCF11 length-curriculum transfer tells us whether stability pacing is specific to tensor-power recurrence or a general tool for tensor-decomposed recurrent models. The headline claim does not depend on that transfer succeeding.