Recurrent networks are usually asked to do one of two opposite things. Machine-learning work on non-normal recurrent networks uses hidden feedforward structure in the connectivity to keep information alive for hundreds of steps. Here "non-normal" means weight matrices whose eigenvectors are not orthogonal, so activity can grow for a while before it decays. Neuroscience work on balanced amplification uses strong excitation cancelled by strong inhibition to erase information quickly, so the circuit can follow a fast-changing input without lagging. We propose that these are not competing designs but two ends of one structure. The recurrent matrix holds a set of strongly damped "balanced" modes that track the input quickly, and a set of slowly decaying modes that store it. A large one-way coupling from the fast modes to the slow ones copies and amplifies what is being tracked into memory. Because the coupling is feedforward rather than feedback, it boosts the stored signal without slowing the tracking modes.
The concrete bet is a recurrent network whose weights are written in Schur form: an orthogonal change of basis applied to a matrix of eigenvalues plus a strictly triangular matrix of one-way couplings. In this form, the decay rate of the tracking modes, the decay rate of the memory modes, and the strength of the fast-to-slow coupling become three separate knobs. The point of separating them is to break a trade-off that the cited literature shows is otherwise forced. In a normal network, high gain requires eigenvalues near instability, and the network becomes sluggish. In a purely tracking-optimized balanced network, memory is discarded by design. We test the idea on tasks that demand both abilities at once, and on standard memory benchmarks where non-normal recurrent networks already have reported numbers.
Core Claim
Balanced amplification is a specific non-normal motif. In a two-population excitatory–inhibitory rate model, the "difference" pattern decays very fast and drives the slower "sum" pattern through a large feedforward weight. Fast and richly structured activity in cortical networks with local inhibition and Non-normal amplification in random balanced neuronal networks analyse this motif. The second paper uses Schur decomposition to show that the gain comes from the triangular part rather than from eigenvalues near zero.
Write the linear dynamics as $\dot{x} = -x + Wx + u$ with $W = P(\Lambda + T)P^\top$. Here $P$ is orthogonal, $\Lambda$ holds eigenvalues and $T$ is strictly upper triangular. Consider one fast mode $a$ with decay rate $b = 1-\lambda_f \gg 1$, one slow mode $m$ with decay rate $\epsilon = 1-\lambda_s \ll 1$, and a coupling $w = T_{ma}$. A unit pulse into $a$ produces
$$m(t) = \frac{w}{b-\epsilon}\left(e^{-\epsilon t} - e^{-b t}\right),$$
which follows from integrating the two-mode triangular system directly. Mode $a$ follows input on the compressed timescale $1/b$, the $\tau/b$ scaling derived in A time-resolved theory of information encoding in recurrent neural networks. Mode $m$ holds an amplified copy for a time of order $1/\epsilon$, with gain set by $w/b$.
Tracking speed depends only on $b$. Memory lifetime depends on $\epsilon$. Write-in strength depends on $w/b$. A normal network cannot separate these, because it has no $w$: its only route to gain is pushing eigenvalues toward 1, which slows everything. The claim is that a recurrent network whose fast tracking modes feed its memory modes through large triangular coupling sits on a strictly better tracking-versus-memory curve than normal, orthogonal, or unconstrained alternatives at matched size. It also predicts that tracking error is flat in $w$, while memory signal-to-noise ratio rises with $w$ until saturation.
Relation to Prior Work
Two machine-learning papers supply the method and motivate the memory side.
Non-normal Recurrent Neural Network (nnRNN) parametrizes recurrent weights in exactly this Schur form. It keeps eigenvalues near the unit circle and learns $T$, and it reports gains over orthogonal networks on character-level Penn Tree Bank. It never places some eigenvalues far inside the stable region to create fast modes. It also never asks for fast tracking, and on Copy it learned $T$ near zero.
Improved memory in recurrent neural networks with sequential non-normal dynamics shows that feedforward chains raise signal-to-noise ratio under noise and nonlinearity. On Copy-500 the chain succeeded in 46 of 462 runs against 0 for orthogonal models. Our slow block reuses that chain idea. What we add is a fast, strongly damped front end whose output feeds the chain.
Four neuroscience papers motivate the tracking side and constrain the design.
- Primate neocortex performs balanced sensory amplification shows that cortex returns to baseline in 20–75 ms and switches rather than sweeps after a stimulus jump. This rules out near-critical and continuous-attractor explanations for sensory gain.
- A time-resolved theory of information encoding in recurrent neural networks shows that the information rate grows with balance tightness. It also explicitly flags the conflict with long memory, and that is the conflict we target.
- Regimes and mechanisms of transient amplification in abstract and biological neural networks acts as a constraint. Imaginary eigenvalue components reduce amplification, so we keep the fast block's eigenvalues real. Networks trained to amplify many inputs collapse to low effective rank, so we measure effective rank of the slow block rather than assume it stays high.
- Fast Sampling-Based Inference in Balanced Neuronal Networks shows the same "break symmetry for speed" logic in sampling. It optimizes a skew-symmetric term offline with a quasi-Newton optimizer (L-BFGS), whereas we learn the coupling by gradient descent inside a task.
None of these papers trains a single network to track and remember at once. None measures how the fast-to-slow triangular coupling moves the trade-off.
Research Program
We will build a "balanced Schur recurrent network" in discrete time, with $h_t = \sigma(P(\Lambda+T)P^\top h_{t-1} + Ux_t)$ and $P$ orthogonal via the exponential parametrization used by nnRNN. The constraints are as follows:
- $\Lambda$ is split into a fast block, whose real eigenvalues lie in a band well inside the unit disk (the discrete-time analogue of large $b$), and a slow block near the unit circle.
- $T$ is restricted to fast-to-slow coupling and within-slow chains.
- Slow-to-fast feedback is zeroed, which keeps the fast modes fast.
- The readout can draw on both blocks.
The fast band, the slow band and the norm of the coupling block $\|T_{sf}\|$ are hyperparameters swept explicitly, so the trade-off can be mapped rather than only optimized.
We pair this with an analysis tool. We Schur-decompose any trained recurrent matrix, including LSTM recurrent blocks linearized at operating points, and report three quantities:
- the fraction of feedforward norm flowing from strongly damped to weakly damped modes;
- the imaginary spread of each block;
- the effective rank of slow-block responses.
This lets us ask whether unconstrained training finds the motif on its own when the task demands both abilities. That outcome is informative either way.
We will also test whether the gain survives saturating nonlinearities. All theory, including the equation above, is linear, and the file identifies nonlinear theory as incomplete. We therefore measure $w$-dependence empirically with $\tanh$ units, and do not claim a theorem.
Baselines
The naive reference is an orthogonal recurrent network (expRNN) plus a plain RNN with identity initialization. It shows what norm preservation alone gives. On character-level Penn Tree Bank, nnRNN reports expRNN at 1.49 bits per character and EURNN at 1.61 (nnRNN).
The strongest existing method on this question is the unconstrained nnRNN. It has the same Schur machinery and the same parameter count, so any win must come from the fast/slow constraint and the coupling, not from the parametrization. It reports 1.47 bits per character on Penn Tree Bank, the number our model must match on that benchmark to show the constraint costs nothing on standard language modeling.
On Copy-500, the reference is the feedback chain of Improved memory in recurrent neural networks with sequential non-normal dynamics at 46 of 462 successful runs. We report success counts under the same protocol.
An LSTM at matched parameter count is included as the practical default.
A stacked two-module baseline is also included: a leaky fast recurrent network feeding a slow one, with unconstrained cross-weights. This tests whether simple hierarchy, rather than Schur-controlled balanced coupling, explains any gain.
Two ablations isolate the key ingredient:
- Zero coupling. Setting $T_{sf}=0$ leaves fast and slow modes but no amplified write-in.
- Normal amplification. Removing $T$ entirely and instead moving fast eigenvalues toward the unit circle to recover the same gain. This tests the critical-slowing alternative directly.
All models share data, hidden size or parameter budget, optimizer, and an equal number of hyperparameter trials, run with the same seeds. Beating the field means two things on the joint task. First, the full model reaches lower tracking error at equal memory accuracy than every baseline. Second, it attains a Pareto front that dominates both ablations.
Why not just…?
Why not just use an orthogonal recurrent network? Orthogonal matrices preserve norms but cannot amplify, and the cited evidence shows this fails under noise: 0 of 462 Copy-500 successes (Improved memory in recurrent neural networks with sequential non-normal dynamics). Their eigenvalues all sit on the unit circle, so they have no fast-decaying modes for tracking. The expRNN baseline measures this directly on the joint task.
Why not just tune eigenvalues near instability for gain? The prediction is that this buys gain at the price of tracking speed, exactly the critical slowing documented in Non-normal amplification in random balanced neuronal networks. It is also the regime that primate MT data rule out (Primate neocortex performs balanced sensory amplification). The normal-amplification ablation tests this. If it matches our model's Pareto front, the core claim is false.
Why not just stack a fast network on a slow one, or use an LSTM? This is the real threat. A block-triangular Schur matrix is mathematically a fast module feeding a slow one in a rotated basis, and an LSTM can learn per-unit timescales through gating. We do not argue around this. The stacked and LSTM baselines are run at matched budget.
- If an unconstrained stacked network matches our model, the contribution narrows to the analysis tool. We would then check whether its learned weights show strong damped-to-slow feedforward coupling, which would mean the motif is what training finds anyway.
- If it underperforms, the eigenvalue banding and the coupling control are doing work that free cross-weights do not discover under equal tuning.
Why not just train the unconstrained nnRNN on the joint task? It may find the motif by itself. Our analysis tool checks whether its trained Schur form develops fast real modes coupled into slow ones. The prior report that it learned $T$ near zero on Copy suggests it may not without the constraint.
Proposed Experiments
The primary task joins tracking and memory, because the file contains no existing benchmark that demands both.
- Tracking. An Ornstein–Uhlenbeck input, of the kind analysed in A time-resolved theory of information encoding in recurrent neural networks, must be reported at every step. We sweep its correlation time down to a few steps.
- Memory. Interleaved cue tokens must be recalled after a delay of 100–500 steps.
- Measurements. Tracking is measured by mean squared error and by lag at peak cross-correlation. Memory is measured by recall accuracy.
- Sweep. We sweep $\|T_{sf}\|$ and the fast eigenvalue band and plot the Pareto front for every model.
No reported result was found on Lacuna for any model on this joint task, so all of its numbers will be new.
The decisive checks are three predictions from the two-mode equation:
- tracking lag stays flat as $\|T_{sf}\|$ grows;
- memory signal-to-noise ratio rises roughly in proportion to $w/b$ until the nonlinearity saturates;
- the normal-amplification ablation shows tracking lag growing with gain.
If tracking lag rises with coupling, the separation claim fails.
Secondary checks place the model on established benchmarks:
- character-level Penn Tree Bank, against nnRNN's 1.47 bits per character;
- Copy-500, against 46 of 462 successful runs;
- permuted sequential MNIST, where the file quotes no nnRNN number, so we report against our own reruns.
We additionally report effective rank of slow-block responses as the number of stored cues grows. This tests whether the low-rank collapse found in Regimes and mechanisms of transient amplification in abstract and biological neural networks caps how many items the amplified write-in can store.
References
- Fast and richly structured activity in cortical networks with local inhibition
- Non-normal amplification in random balanced neuronal networks
- A time-resolved theory of information encoding in recurrent neural networks
- Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
- Improved memory in recurrent neural networks with sequential non-normal dynamics
- Primate neocortex performs balanced sensory amplification
- Regimes and mechanisms of transient amplification in abstract and biological neural networks
- Fast Sampling-Based Inference in Balanced Neuronal Networks