Reading Across Layers: Cross-Layer State Retrieval for Linear-Attention Language Models

Linear attention replaces the quadratic all-pairs comparison of standard attention with a running summary. Each layer keeps one fixed-size matrix, its "state," that accumulates key–value products as tokens arrive, and each new token reads from that matrix with its query. This makes generation cost constant per token, but it also creates a capacity ceiling. A fixed matrix can only store so many distinguishable associations. Once it fills up, old facts collide and become unrecoverable, and retrieval collapses: a linearized Llama scores 0.6% on a simple needle-in-a-haystack test at 4K tokens (LoLA). Current fixes either enlarge or reshape the state inside one layer, or bolt on a growing cache of exact tokens, or reinsert expensive softmax layers.

We point out something these fixes ignore. A deep linear-attention model already holds $L$ separate states in memory at every decoding step, one per layer. Each was written with different key features, so its collisions fall in different places. Yet each layer reads only its own state. We propose letting every layer also query the states of a small window of earlier layers, through its own query and a learned, input-dependent weight over source layers. This multiplies the associative memory each read can reach by the window size, and it adds no decode-time memory at all, because those states are kept anyway. The added cost is a few extra small matrix–vector products per token. The core claim is concrete. At equal decode memory, cross-layer state reads recover a large part of the long-context retrieval that linear attention loses, without the growing cache of hybrid or sparse-cache designs.

Core Claim and Mechanism

Let $S^{(j)}_t \in \mathbb{R}^{d_k \times d_v}$ be the gated recurrent state of layer $j$ after token $t$, as in Gated Linear Attention. In standard GLA, layer $l$ outputs $\phi(q^{(l)}_t)^\top S^{(l)}_t$. We replace this with

$$o^{(l)}_t = \sum_{j \in \mathcal{W}(l)} a_{lj}(x_t)\, \phi\big(q^{(l,j)}_t\big)^\top S^{(j)}_t,$$

where $\mathcal{W}(l)$ is a window of $W$ source layers (layer $l$ itself plus $W-1$ earlier layers), and $a_{lj}(x_t)$ is a softmax over those sources computed from the current hidden state $x_t$. The query $q^{(l,j)}_t$ is a source-specific query, built as a shared projection plus a per-source diagonal rescaling so that parameters grow only slightly.

The design choice that matters is reading each source state with its own query rather than reading one mixed state. A weighted sum $\sum_j a_{lj} S^{(j)}$ is still a single $d_k \times d_v$ matrix, so it inherits the same rank ceiling that Breaking the Low-Rank Dilemma of Linear Attention measured (rank about 28 for linear attention against about 56 for softmax). Separate reads behave like attention over a key space of width $W d_k$. The output can therefore reach rank up to $\min(W d_k, d_v)$, and the reachable stored associations scale with $W$.

Causality in depth is preserved: layer $j<l$ has already been computed when layer $l$ runs, both in chunkwise-parallel training and at decode. The window $W$ caps the cost at $O(W)$ reads per layer. This avoids the quadratic-in-depth blowup that caused out-of-memory failures in full layer attention (MRLA).

Relation to Prior Work

Two kinds of work motivate the gap. LoLA argues that rank-bounded states make memory collisions inevitable and adds a per-layer sparse cache of exact tokens, chosen by a Self-Recall Error score (how badly the state reconstructs a token's value). That cache grows with context, and LoLA explicitly does not share information across layers. RALA raises rank inside a single block through query-weighted buffers and an element-wise output modulation, and was tested only on vision models of about 100M parameters. We leave the block untouched and get capacity from depth instead. The two are compatible, and we test whether they stack.

Three cross-layer works supply the mechanism, and each differs from ours in a specific way. MRLA treats layers as tokens and lets deeper layers attend to earlier feature maps in CNNs and vision transformers. It aggregates outputs, not recurrent memories, and never touches causal language models. SoLA-Vision injects early linear-layer hidden states into later softmax layers on a fixed stage schedule, in non-causal vision models. CLAA averages attention scores uniformly across a window of layers to decide which prompt tokens to prune during prefill. It uses softmax backbones and no learned weights.

The classic weighted-sum aggregation line also bears on the design. Wu et al. and Chen et al. learn scalar weights over layer outputs, PCLM conditions those weights on a prompt, and Value Aggregation pools attention value vectors across 50–85% depth for embeddings. All of these combine finished representations of the current token. None can retrieve a past token that one layer's state has forgotten but another layer's state still holds. That retrieval is our target.

Two results constrain the design. SoLA-Vision proves that stacking linear layers grows the effective receptive field only as $O(\sqrt{M})$. Our reads do not change what each state stores, so if the needed information was never written into any state, cross-layer reads cannot help. The diagnostic below tests for exactly this. The second constraint is a tension in the literature. CLAA finds early layers "blind" for judging token importance, while MRLA finds deep layers attend most to the first layer. We therefore learn the source weights rather than hard-coding a window of adjacent layers, and we report which sources the model actually uses.

Research Program

Pre-check before building. On a trained GLA checkpoint, we compute LoLA's Self-Recall Error in two ways. The first reconstructs each evicted token's value from its own layer's state only. The second uses a least-squares readout fitted over the union of $W$ neighboring states. If the union does not lower recall error substantially, the states are redundant and the idea fails cheaply.

Training from scratch. We train GLA models at 340M and 1.3B parameters on FineWeb-Edu (SlimPajama as a replication corpus) for 15B and 50B tokens. Each model gets cross-layer reads with $W \in \{2,4,8\}$. We extend the FlashLinearAttention chunkwise kernel so that layer $l$'s intra-chunk step also reads the inter-chunk states of its source layers, which are already in high-bandwidth memory.

Retrofit track. We add cross-layer reads to a linearized 8B model in the LoLCATs style and fine-tune briefly. This allows a direct comparison with LoLA on its own ground.

Analyses. We measure output effective rank as RALA does, map the learned source-weight patterns across depth, and profile decode throughput and memory.

Baselines

The naive reference is the same GLA architecture trained on identical data, tokens, and tuning budget, reading only its own state. For the linearized track, the naive point is the base linear model, which LoLA reports at 0.6% on RULER S-NIAH-1 at 4K.

The strongest existing method on long-context retrieval is LoLA itself, which reaches 97.4% on RULER S-NIAH-1 at 4K with a cache 4.6x smaller than Llama-3.1-8B's. We run it on the identical linearized backbone with the same fine-tuning data. We set our model's state budget so that its total decode memory equals LoLA's at 4K, and we then compare at 8K and 16K, where LoLA's sparse cache keeps growing and ours does not. Beating LoLA means matching it within a few points at 4K and exceeding it per byte of decode memory at longer contexts. We do not expect to beat it at unlimited memory.

For short-context quality in the from-scratch track, the reference number in the file is FiX. It reports 55.24% average over eight commonsense tasks at 340M–760M scale with a softmax-family model. The file contains no reported result on Lacuna for GLA at matched scale on that suite, so we retrain GLA ourselves.

Three ablations isolate the key ingredient. The first is a mixed-state read, where one query reads $\sum_j a_{lj} S^{(j)}$. If this matches separate reads, the rank argument is wrong. The second is a static scalar softmax weighting over layer outputs in the style of Wu et al.. This is the ordinary layer-aggregation answer. The third fixes source weights to uniform over adjacent layers, as in CLAA. This separates the value of learned, input-conditioned routing.

Why not just…?

Why not just make each layer's state $W$ times larger? A larger $d_k$ also raises capacity, and MHLA-style head mixing recovers diversity within a block. But it multiplies decode memory and per-token state updates by $W$. Our reads reuse states that already exist. We train a GLA with $W\times$ key dimension and compare at both matched FLOPs and matched decode memory. If it wins at matched memory, our idea adds nothing beyond a cheaper capacity knob, and we will report that.

Why not just interleave a few softmax or sliding-window layers? This is the real threat. SoLA-Vision's schedule and LoLA's sliding window both show that exact attention fixes retrieval. We train a hybrid with one softmax layer per four at matched parameters and compare on RULER and BABILong at matched decode memory. The hybrid's KV cache grows linearly with context, so we expect it to win at short contexts and lose per byte beyond 8K. If the hybrid wins even at matched memory, the proposal is not worth pursuing.

Why not just learn a weighted sum of layer outputs? This is the standard aggregation tool, and it helps classification (Chen et al. lose 1.27 points on SNLI without it). But it combines the current token's representations. It cannot retrieve a past token that this layer's state has lost. The static-output ablation should match our method on perplexity and commonsense tasks and fall well short on needle retrieval. If it does not, retrieval gains come from optimization effects rather than memory.

Decisive Checks

The idea stands or falls on three results. First, the pre-check must show that union-state recall error is clearly below own-state error. Second, at 1.3B scale and matched decode memory, cross-layer reads with $W=4$ must beat the $W\times$-wider state and the static-output ablation on RULER single- and multi-key needle tasks at 4K–16K and on BABILong at 1K–16K. Third, the mixed-state ablation must fall clearly behind separate reads, which confirms that the gain comes from expanded reachable rank and not from extra parameters.

We additionally require perplexity on C4 and PG19 and the commonsense average to be no worse than GLA, and decode throughput to be within 15% of GLA. Finally, we report whether learned source weights concentrate on early layers (MRLA's pattern) or on nearby layers (CLAA's assumption). This bears directly on the open disagreement between those two papers.