Heavy-Tailed Input Weights for Locating Transitions in Bidirectional Recurrences

Many sequence problems are mostly about finding when something changes. Examples are the moment a sleeping mouse enters REM sleep, the week an outbreak turns, or the cycle at which a battery starts to degrade. Bidirectional recurrent models read a sequence both forward and backward, so they are well suited to these problems: at each step they know what came before and what comes after. Separately, work on heavy-tailed weight initialization has found something useful. Heavy-tailed distributions are ones where a few weights are orders of magnitude larger than the rest. When weights are drawn this way instead of from a Gaussian, the network forms a sparse "skeleton" of very strong connections. That skeleton can keep recurrent dynamics robust and can capture sharp jumps that Gaussian-style models smooth over. Nobody has put the two together.

We propose a concrete form of that union. In a bidirectional linear recurrent model, the recurrent part stays bounded and stable. We draw only the matrices that feed signals into the recurrence and read them out from heavy-tailed distributions. Our hypothesis is that the resulting few high-gain channels act as built-in jump detectors. A forward pass and a backward pass that both contain such channels should then pin down the location of an abrupt transition more precisely than either direction alone. The test is an interaction: heavy tails should help bidirectional models more than causal ones, and help more on transition-heavy tasks than on smooth ones. If the benefit is just a generic optimization effect, no interaction will appear.

Core Claim

The obvious way to combine these directions would be to make the recurrent weight matrix heavy-tailed, as done for unidirectional networks. That route runs straight into a stability problem. The bidirectional Linear Recurrent Unit model BLUR (Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion) keeps every eigenvalue of its diagonal state transition strictly inside the unit circle. That property is what makes its fast parallel scans over long sequences safe. An unconstrained heavy-tailed matrix would put eigenvalues far outside that circle. We avoid the conflict by placing the tails elsewhere. Each direction of the model computes

$$h^{f}_t = \lambda \odot h^{f}_{t-1} + B_f x_t, \qquad h^{b}_t = \lambda' \odot h^{b}_{t+1} + B_b x_t,$$

where $x_t$ is the input at step $t$, $h^f_t$ and $h^b_t$ are the forward and backward hidden states, $\lambda$ and $\lambda'$ are diagonal decay factors kept inside the unit circle, and $B_f, B_b$ are input projections. The two states are combined by a readout and a gated fusion layer. We keep $\lambda, \lambda'$ exactly as in BLUR and draw the entries of $B_f$, $B_b$, and the readout from a symmetric $\alpha$-stable distribution with $\alpha \in (1,2)$. The parameter $\alpha$ sets tail weight: $\alpha = 2$ is Gaussian, and smaller values give heavier tails. Because the transition is a stable contraction, each state is a geometrically weighted sum of heavy-tailed projected inputs. The states can therefore have heavy-tailed magnitudes, but they cannot diverge.

The claim we test is specific. Heavy-tailed projections improve how precisely transitions are located in bidirectional linear recurrences. The improvement comes from a small set of outlier weights, it persists through ordinary end-to-end training without any per-step projection of the weights, and it is larger for bidirectional models than for causal ones.

Why This Is Grounded

Three lines of evidence motivate the hypothesis. First, deep kernel models with infinite-variance prior weights capture step discontinuities that Gaussian processes oversmooth (Deep Kernel Posterior Learning under Infinite Variance Prior Weights). That result concerns static regression, not sequences. Second, in heavy-tailed recurrent reservoirs, the top 1–3% of weights govern the network's dynamics, and pruning them destroys the network's robustness (Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks). That work used fixed recurrent weights, one direction, and no end-to-end training. Third, heavy-tailed connectivity in trained cognitive-task networks forms an effectively low-rank structure that acts as a gate and supports generalization outside the training range (Biologically plausible heavy-tailed connectivity enhances generalizations on cognitive tasks in recurrent neural networks). However, that work enforced the distribution at every training step using an optimal-transport projection.

On the bidirectional side, two papers argue that future context matters precisely for transitional states. The mouse sleep-staging study (Evaluating the Influence of Temporal Context on Automatic Mouse Sleep Staging through the Application of Human Models) makes this argument for REM onset, and FIGI-Net (Fine-Grained Forecasting of COVID-19 Trends at the County Level in the United States) makes it for outbreak inflections. Both use Gaussian-initialized bidirectional LSTMs.

Two results constrain the design. The theory for $\alpha$-stable ReLU networks shows that heavy tails need an extra $(m\log m)^{-1/\alpha}$ width scaling, where $m$ is layer width, to avoid blow-up (Large-width asymptotics and training dynamics of $\alpha$-Stable ReLU neural networks). We use this scaling for $B$ and the readout. Initializations engineered purely for signal propagation can also overfit (Gaussian Pre-Activations in Neural Networks: Myth or Reality?), so we report held-out results only. The fractional-moment scheme of Fractional moment-preserving initialization schemes for training deep neural networks supplies a second recipe for choosing scale. It gives an alternative to the stable-law scaling that controls the order-$s$ moment, for $s<2$, rather than the variance.

There is also a confound to measure rather than assume away. SGD itself pushes trained weights toward heavy tails (Weight Space Probability Densities in Stochastic Learning: I. Dynamics and Equilibria; Asymptotics of Gradient-based Neural Network Training Algorithms). Gaussian-initialized models may therefore end up heavy-tailed anyway, so initialization could matter only for speed.

Research Program

We will build a single BLUR-style codebase in which only the initialization of $B_f$, $B_b$, the readout, and the fusion weights varies. It will have a causal variant that drops the backward pass and doubles the forward state size, so that parameter counts match. We sweep $\alpha \in \{1.2, 1.5, 1.8, 2.0\}$. Throughout training we track three things: the tail index of the projection weights, measured with a Hill estimator, which estimates tail weight from the largest values; the identity and stability of the top 1–3% of weights; and the effective rank of the projections. This tracking answers the open question raised in Biologically plausible heavy-tailed connectivity enhances generalizations on cognitive tasks in recurrent neural networks: does initialization alone preserve the skeleton without continuous constraints?

For the mechanism, we will identify which channels respond most strongly at true transitions. We then test whether those channels are the ones fed by outlier weights. Finally, we check whether the forward and backward passes bracket each transition, with the forward response just after it and the backward response just before it.

As a secondary arm, we put heavy tails into the timescale spectrum instead. Here the decay rates $1-|\lambda|$ are drawn from a power law, while the unit-circle constraint still holds. This lets us separate "a few strong inputs" from "a few very long memories."

Baselines

The naive reference point is BLUR with its standard bounded initialization of eigenvalues and Gaussian/Glorot projections. The same architecture with Kaiming-scaled projections plus layer normalization serves as a second "do nothing special" control. This pair represents what a practitioner would actually run.

The strongest published system on our main benchmark is L-SeqSleepNet, a bidirectional LSTM sleep stager. It reports 95.3% balanced accuracy on the Kornum mouse dataset and 96.3% on the Brown dataset, with REM F1 near 90% in some cohorts (Evaluating the Influence of Temporal Context on Automatic Mouse Sleep Staging through the Application of Human Models). We will retrain it on our exact splits and context window, using the paper's 28–44 second sweet spot, and with the same hyperparameter-search budget. For long-horizon forecasting, the reference is BLUR itself. The file reports that it beat Informer and PatchTST, but no reported MSE number was found on Lacuna for BLUR on ETT, Weather, or ECL, so we will establish that number ourselves.

Fairness rules apply to every comparison. Heavy-tailed weights at $\alpha<2$ have infinite variance, so we do not match variance. Instead, we match the median absolute hidden-state magnitude over a fixed batch at initialization. Every arm gets the same learning-rate sweep, here eight values, and ten seeds. Parameter counts and training steps are identical across arms.

Three ablations isolate the key ingredient. First, we prune the top 2% of projection weights at initialization, which tests whether the outliers carry the effect. Second, a spike-and-slab initialization combines a Gaussian with a small random set of large entries of matched magnitude. This tests whether the continuous tail shape matters or only the existence of a few large weights. Third, the causal variant at each $\alpha$ measures the direction-by-initialization interaction. Beating the baselines means two things. Heavy-tailed bidirectional models must reduce transition-localization error with a seed-level confidence interval excluding zero, while matching or improving balanced accuracy. They must also show a larger gain over Gaussian initialization than the causal variant shows.

Why not just…?

Why not just use Kaiming or Xavier initialization with layer normalization? Normalization fixes signal scale. It does not create a sparse set of high-gain channels, and our hypothesis concerns that structure, not scale. The early-training gains of fractional-moment schemes over Kaiming (85.02% vs 80.08% early training accuracy on a 20-layer MNIST network, Fractional moment-preserving initialization schemes for training deep neural networks) could be purely an optimization-speed effect that normalization also delivers. That is a real threat. It is why we report final held-out transition metrics, run the normalized baseline as a primary control, and look for the interaction with directionality, which a speed effect would not produce.

Why not just add $\ell_1$ sparsity or prune a Gaussian model? In unidirectional cognitive-task networks, heavy-tailed models outperformed $\ell_1$-regularized ones on out-of-range generalization (Biologically plausible heavy-tailed connectivity enhances generalizations on cognitive tasks in recurrent neural networks). However, that comparison used constrained training, not initialization alone, so it does not settle our setting. We include an $\ell_1$ arm. The spike-and-slab ablation is the sharper version of this objection. If spike-and-slab matches $\alpha$-stable, we will report that the active ingredient is "a few large input weights" and not the tail law. That would be a simpler and still useful result.

Why not just use an explicit low-rank projection? Heavy tails reach an effective rank similar to that of forced low-rank models (Biologically plausible heavy-tailed connectivity enhances generalizations on cognitive tasks in recurrent neural networks). On the other hand, removing the outliers from a full-rank background destroyed robustness in reservoirs (Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks). A rank-matched factorization $B = UV^\top$ run as its own arm decides which matters for transition localization.

Why not just tune the eigenvalue ring or use log-spaced timescales? Spreading the timescales may give long-memory channels that help with transitions without any heavy-tailed projections. Our secondary timescale arm and a log-uniform timescale control test this directly. If timescale tuning alone matches the projection effect, the projection claim fails.

Why not just use bidirectional attention? Attention is the natural choice for short sleep-staging windows. We include a size-matched non-causal Transformer encoder as a reference. Our claim, however, concerns linear-time recurrences on long sequences, where DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows and BLUR document large throughput advantages over attention.

Proposed Experiments

The first experiment is a synthetic benchmark we construct. It consists of piecewise-constant multichannel signals with random jump locations and sizes, embedded in either Gaussian or heavy-tailed noise. We measure absolute error in localizing each jump and the rate of falsely detected jumps. This is where the mechanism analysis happens. If outlier-fed channels do not bracket jumps here, the hypothesis is wrong.

The second experiment uses the Kornum and Brown mouse sleep datasets, subject to access. We report balanced accuracy, REM F1, and a transition-window F1 computed only on epochs within ±2 epochs of a stage change. We also report the epoch-distance error between predicted and true stage changes. The decisive check is the 2×4 grid of {causal, bidirectional} × {α}: the bidirectional gain from heavy tails must exceed the causal gain.

The third experiment uses ETT and Weather long-horizon forecasting as a smooth-signal control. Here we predict no gain, and possibly a small loss, consistent with the collapse to low-dimensional dynamics reported in Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks. A uniform gain across all tasks would point to optimization, not transition detection.

As an optional fourth test, we will apply the same comparison to the NASA B0005 battery data. There CNN-BiLSTM-AM reports RMSE 0.00487 at a 90% prediction start point (State-of-Health Prediction of Lithium-Ion Batteries Based on CNN-BiLSTM-AM), and the abrupt capacity-regeneration jumps offer a real-world transition test.

The idea is falsified under any of three outcomes: the pruning ablation leaves the gain intact, the causal and bidirectional gains are equal, or Gaussian-initialized models converge to the same tail index and the same localization error by the end of training.