A recurrent model, or a linear-attention model that runs like one, carries a fixed-size summary of everything it has read. That summary is its "state." The state size sets both how much the model can remember and how much memory it needs at inference. The cost is steep. The matrix that updates the state from one step to the next has one weight for every pair of state entries, so doubling the memory roughly quadruples those parameters. Because of this memory-parameter coupling, the state size is chosen once, before training, and then fixed for good. A deployment that needs a smaller memory footprint must train a separate, smaller model.
We propose to train a single recurrent layer whose state is nested. The state is split into ordered blocks. Each block may read from the blocks before it but never from the blocks after it. Readout heads attach to every prefix of blocks, and each head predicts a residual correction on top of the previous one. Because no block depends on later blocks, simply dropping the trailing blocks at inference yields a valid, smaller recurrent model. Supervising every prefix pushes the most useful information into the leading blocks and finer detail into the later ones. Put simply, we take multi-scale deep supervision (attaching extra losses to intermediate parts of a network) and apply it across the width of the memory rather than across the depth of the network. The result is a single trained model whose memory footprint can be chosen after training, with one shared set of weights serving every size.
Core Claim
We claim that a block-lower-triangular state transition, trained with residual, coarse-to-fine losses on every prefix of its state, produces a model with three properties. Each truncated prefix should come close to the quality of a model trained from scratch at that state size. The full-size model should lose little or nothing relative to terminal-only training. The shared weights cost far fewer total parameters than training one model per size.
The parameter argument is simple. Take a dense recurrent state of width $h$ split into $r$ equal blocks. A block-lower-triangular transition matrix has about $h^2/2$ entries and serves all $r$ sizes. Separately trained models of widths $h/r, 2h/r, \dots, h$ together need $\sum_{k=1}^{r} (kh/r)^2 \approx h^2 r/3$ transition parameters. The nested model is therefore roughly $2r/3$ times cheaper than keeping a separate model per size.
The scientific question is whether the auxiliary losses actually create a usable ordering of memory, and whether they do so without the harms that earlier work documented for intermediate supervision.
Where This Sits
The proposal borrows its supervision method from two depth-wise techniques. T-MLP attaches output "tails" to every hidden layer of a coordinate network. Each tail adds a residual to the previous prediction, which forces early layers to encode low-frequency content and later layers to encode detail. We reuse the residual-tail form, but place the tails over blocks of recurrent memory rather than over layers. Deep Trajectory Supervision shows that asking shallow layers for confident one-hot predictions creates an "impedance mismatch." That paper replaces hard targets with a schedule that rises from low to high confidence, and gains +2.2% Top-1 on CIFAR-100 and +0.8% on ImageNet-1K. We adapt this schedule to width: small prefixes receive softened targets and the full state receives the hard target.
The architecture we modify is the partitioned linear-attention memory of Head-in-Head in Linear Attention. That paper splits each head's memory into $r$ blocks and mixes them with a learned $r \times r$ matrix, which raises the rank of the state update from 1 to $r$. Its mixing is all-to-all and its training uses only the terminal language-modeling loss, so no prefix of its memory is meaningful on its own. We constrain the mixing matrix to be lower-triangular and add prefix supervision. For classical recurrent networks, Pyramidal Recurrent Unit already splits the context vector into groups with independent linear maps. We change those independent maps to triangular group-to-group maps.
Three works on compressing the transition weights play a different role. Kronecker Recurrent Units, Tensor-Train RNNs and FDHT-LSTM all attack the coupling by factorizing the transition, but each produces a single fixed memory size trained with a terminal loss. Our approach is complementary: it does not factorize the transition, it amortizes it across sizes.
One paper constrains the design. When Intermediate Supervision Doesn't Help found that supervising recurrent steps with prescribed algorithmic traces led to collapse to at most 0.1% accuracy under topology shift. Our auxiliary losses use only the final task target and never prescribe intermediate content. Even so, that result obliges us to test whether prefix supervision harms extrapolation.
The closest bridging work, DB-MSMUNet, combines a state-space encoder with deep supervision on convolutional decoders. It does not touch the structure or size of the recurrent state.
Research Program
We will build nested variants of two architectures: a Gated DeltaNet layer with Head-in-Head partitioned memory, and a small grouped recurrent network in the style of the Pyramidal Recurrent Unit.
Let the state at step $t$ be a stack of blocks $S_t^{1}, \dots, S_t^{r}$. The update of block $i$ may depend on blocks $j \le i$ only. For each prefix $k$, a tail head produces $y^{(k)} = y^{(k-1)} + g_k(S_t^{1..k})$, with $y^{(0)}=0$. Training minimizes
$$ \mathcal{L} = \sum_{k=1}^{r} w_k \, \mathrm{CE}\big(y^{(k)}, \tilde{z}_{\epsilon_k}\big), $$
where $\mathrm{CE}$ is cross-entropy, $\tilde{z}_{\epsilon_k}$ is the target smoothed by label-smoothing strength $\epsilon_k$, and both $\epsilon_k$ and the weights $w_k$ follow a schedule. Smoothing starts high for $k=1$ and reaches zero at $k=r$, mirroring the depth schedule of Deep Trajectory Supervision. The tails add little compute, because every prefix shares the same forward pass through the recurrence.
The analysis asks three questions. First, does information order itself as intended? We will probe each block for long-range facts, such as recall of earlier spans, versus local detail, such as next-token lexical identity. Second, how does quality fall as blocks are dropped, and does that curve differ between recall-heavy and reasoning-heavy tasks? Third, does the triangular constraint on its own, without supervision, already produce an ordering?
Baselines
The naive reference point is post-hoc truncation of an ordinary model. We train a standard Head-in-Head Gated DeltaNet with a terminal loss and all-to-all mixing, then zero the trailing memory blocks at inference and refit only a linear readout. We expect this to degrade badly, because nothing makes the leading blocks self-sufficient. It sets the floor that any nested method must clear.
The strongest competitor for truncated quality is a family of independently trained models, one per state size $k \cdot h/r$. Each uses the same architecture, the same data and token budget, and the same learning-rate sweep. This is the quality upper bound at each size, and it pays the full summed parameter and training cost. We count nested training as beating the practical frontier if every prefix comes within a small, pre-registered gap of its independent counterpart (we will fix the tolerance from seed variance before running) while using one training run.
For the full-size model, the reference is the published Head-in-Head result. At 1.3B parameters on 100B SlimPajama tokens, HH-GDN reports 52.65 average commonsense accuracy, against 51.76 for Gated DeltaNet, and 37.82 against 36.07 on SQuAD recall (Head-in-Head in Linear Attention). Our budget is smaller, so we will retrain HH-GDN ourselves at our scale with an identical tuning budget. We treat the published figures as context, not as a direct comparison. At matched scale, the full-size nested model counts as not harmed if it stays within seed variance of our retrained HH-GDN.
Two ablations isolate the key ingredient. The triangular-only ablation keeps the lower-triangular mixing but trains with the terminal loss alone, then fits prefix readouts afterwards. This separates what the structure contributes from what the supervision contributes. The hard-target ablation uses one-hot targets with equal weights on every prefix, which tests whether the soft schedule matters for width as it does for depth.
For small-scale recurrent networks, we compare against the grouped recurrence of the Pyramidal Recurrent Unit on PTB and WikiText-2. The file reports only a relative gain of up to 1.3 perplexity over AWD-LSTM (no reported result found on Lacuna for absolute PTB or WT-2 perplexity of PRU), so we will rerun it. On pixel-by-pixel MNIST we also compare against Kronecker Recurrent Units, which reach their best results with 11K parameters. No reported accuracy value was found on Lacuna for KRU, so again we rerun it.
Why not just…?
Why not just train a separate model for each memory size? This is the honest quality ceiling, and in raw per-size accuracy it may win. Its cost is roughly $2r/3$ times more transition parameters, $r$ training runs, and $r$ checkpoints to store and serve. The family-of-models baseline measures exactly how much quality nesting gives up. If the gap exceeds seed variance at most sizes, the proposal's value reduces to cheaper amortization, and we will report it that way.
Why not just compress the state of a normally trained model after the fact, for example by keeping its top principal directions? For linear attention the state is a matrix, so a low-rank approximation at each step is a real alternative. It is a genuine threat, because it needs no retraining. It has two weaknesses. The principal directions change with the input, so the projection must be computed on the fly. The update rule was also never trained to run on a projected state, so errors can compound over long sequences. We will include per-step low-rank projection of the trained HH-GDN state as an extra baseline and compare degradation at 16k-token extrapolation, where compounding error should show most clearly.
Why not just use the triangular structure and skip the auxiliary losses? Causal ordering alone guarantees that truncation is valid, but it does not guarantee that truncation is useful. The model could still place its key recall information in the last block. The triangular-only ablation settles this directly. If it matches the supervised version, deep supervision is unnecessary and the contribution shrinks to an architectural constraint.
Why not just factorize the transition, as Kronecker or tensor-train networks do? Factorization lowers the cost of one fixed memory size. It does not make the size adjustable after training. The two approaches compose: a Kronecker-factored triangular transition is possible. We treat factorization as orthogonal and do not claim to beat it at a fixed size.
Proposed Experiments
The main experiment pretrains nested and baseline Head-in-Head Gated DeltaNet models at a few hundred million parameters on SlimPajama, with $r=4$ and $r=8$ memory blocks per head. For every prefix size, we evaluate language-modeling perplexity, the commonsense suite used by Head-in-Head (Lambada, ARC, HellaSwag, PIQA, WinoGrande), and SQuAD recall. We then extrapolate from 2k training length up to 16k tokens.
The decisive check is a set of curves plotting quality against inference state size for four methods: nested training, the independent family, post-hoc truncation, and low-rank projection. The proposal succeeds if the nested curve lies near the independent family and well above both post-hoc methods, while the full-size point stays within variance of the retrained HH-GDN. We expect recall to degrade under truncation faster than commonsense accuracy does. If that holds, the ordering we induce is meaningful, and the probe analysis should show long-range content concentrated in the early blocks.
Two cheaper settings test generality. The first is PTB and WikiText-2 with a nested grouped recurrent network, which checks the dense-recurrence parameter accounting. The second is pixel-by-pixel MNIST and the length-generalization maze tasks from When Intermediate Supervision Doesn't Help. The mazes explicitly test whether prefix supervision, unlike the trace supervision in that paper, leaves out-of-distribution generalization intact. If nested models collapse on looped mazes where terminal-only models do not, the soft schedule has failed to avoid the harms of intermediate supervision, and that finding would bound where the method can be used.
References
- T-MLP: Tailed Multi-Layer Perceptron for Level-of-Detail Signal Representation
- Deep Trajectory Supervision: Deep Supervision Strikes Back
- Head-in-Head in Linear Attention
- Pyramidal Recurrent Unit for Language Modeling
- Kronecker Recurrent Units
- Tensor-Train Recurrent Neural Networks for Video Classification
- Towards Extremely Compact RNNs for Video Recognition With Fully Decomposed Hierarchical Tucker Structure
- When Intermediate Supervision Doesn’t Help: Evidence from Recurrent CNNs
- DB-MSMUNet: Dual Branch Multi-Scale Mamba UNet for Pancreatic CT Scans Segmentation