Hybrid language models pair a short window of exact attention with a cheap "global" memory that compresses everything older into a fixed-size matrix. That matrix behaves like a lossy code. Each stored fact is written along a direction, and reading one fact back also returns a little of every other fact, weighted by how much their directions overlap. Frame theory, the mathematics of spreading many vectors across few dimensions, has studied exactly this overlap problem for decades. Its answers are sharp. The Welch bound limits how small the worst overlap can be. Equiangular tight frames (ETFs), sets of vectors whose pairwise overlaps are all equal, are the configurations that meet it. ETFs exist only for a sparse, irregular set of sizes.
This note proposes that the global memory should store facts at a fixed set of address directions, chosen in advance as a near-optimal line packing, rather than at directions the network learns freely. The claim is not that such addresses lower average interference; we show below that they cannot. The claim is narrower: they make interference uniform across whichever subset of addresses a given context happens to fill. That is the property that should govern worst-case recall. The sparse existence of ETFs then stops being a curiosity and becomes an architectural design variable that can be tested directly.
Core Claim
Consider a linear-attention memory with address vectors $a_i \in \mathbb{R}^D$ of unit norm and value vectors $v_i$. Writing $M$ items produces the state $S = \sum_{i} a_i v_i^\top$. Reading with the address of item $j$ gives
$$S^\top a_j = v_j + \sum_{i \neq j,\, i \in \mathcal{O}} \langle a_i, a_j \rangle\, v_i ,$$
where $\mathcal{O}$ is the set of occupied addresses. Suppose the $N$ candidate addresses form a tight frame, meaning $\sum_{i=1}^N a_i a_i^\top = \tfrac{N}{D} I$. Then, summed over all $N$ addresses, the interference energy on every row is exactly $\sum_{i \neq j} \langle a_i, a_j\rangle^2 = N/D - 1$, whatever the frame.
Tightness alone therefore fixes the average crosstalk, and random unit vectors land close to the same value in expectation. A real context, however, fills only a subset $\mathcal{O}$ of size $M$. For a general frame, the interference on item $j$ depends heavily on which subset is filled. For an ETF it is always $(M-1)\mu^2$ with $\mu^2 = \tfrac{N-D}{D(N-1)}$, the Welch value, regardless of which subset.
This is the erasure-robustness argument of Grassmannian Frames with Applications to Coding and Communication, which designs frames that minimize worst-case correlation so that losing coefficients does not concentrate error. Here it is transplanted from channel coding to attention memory. The argument also exposes a trade-off. More addresses $N$ make it less likely that two distinct keys fall on the same address; a birthday-style estimate puts the collision rate near $M^2/2N$. More addresses also raise $\mu^2$ toward $1/D$. Together these predict an optimal $N$ for a given $D$ and context load $M$.
The prediction is falsifiable. If addresses are equiangular, the fraction of items recalled in the worst decile of queries should follow this closed-form curve, and it should beat random or learned addresses at matched state size. If addresses are tight but strongly non-equiangular, it should be worse. If equiangular and random addresses perform the same on worst-case recall, the idea is wrong.
Relation to Prior Work
Several hybrid models motivate the gap but leave address geometry uncontrolled:
- Simple linear attention language models balance the recall-throughput tradeoff (BASED) proves that a fixed-size recurrent state caps associative recall. It pairs a Taylor-approximated linear attention with a 64–128-token exact window.
- Long-Short Transformer builds its global stream from input-dependent low-rank projections, with no guarantee on how coherent the projected bases are.
- AllMem and ATLAS replace the linear memory with trained test-time memory networks.
All of these let the network choose its write directions. We fix them, and ask whether the choice matters.
Three papers supply the method:
- Asymptotic Frame Theory for Analog Coding shows that ETFs give compact, predictable noise responses on random subsets of vectors, compared with Gaussian frames. This is the subset-uniformity property we rely on.
- Robust Dimension Reduction, Fusion Frames, and Grassmannian Packings proves that tight frames minimize reconstruction error under white noise. This is the averaged statement that our crosstalk identity reproduces.
- Minimization of the probabilistic p-frame potential shows that for overlap penalties with exponent $p<2$ the minimizers are repeated orthonormal bases. For $p>2$ an ETF is the unique minimizer whenever one exists. This gives us a principled "tight but not equiangular" control.
Construction difficulty constrains the design. ETFs exist only at isolated parameters (Packings in Real Projective Spaces). Exact families come from combinatorial blueprints:
- Steiner and Kirkman designs (Kirkman Equiangular Tight Frames and Codes).
- Steiner triple systems with Hadamard matrices (Tremain equiangular tight frames).
- Hyperovals, including 76 vectors in $\mathbb{C}^{19}$ (Equiangular Tight Frames From Hyperovals).
- Nonabelian groups (Optimal Line Packings from Nonabelian Groups).
We take codebooks from these families where the head dimension permits. Elsewhere we fall back on numerically optimized packings, accepting the warning in An ETF view of Dropout regularization that gradient search on coherence is non-convex and stalls above the Welch bound.
The closest uses of fixed equiangular geometry in deep networks place it at the output layer only. Early Exit with Disentangled Representation and Equiangular Tight Frame fixes BERT exit heads to a simplex ETF. Equiangular Basis Vectors replaces classifiers with fixed spread-out vectors. In both, the number of vectors is the class count and the vectors are never superposed in a shared state. Our setting differs on both counts: there are more addresses than dimensions, and they are superposed in one memory matrix, which is exactly where Welch-bound packing becomes the binding constraint.
Research Program
The architecture is a BASED-style hybrid. It keeps the exact sliding window unchanged and alters only the global stream.
Each key is first mapped to a fixed codebook $A=\{a_1,\dots,a_N\}\subset\mathbb{R}^D$. The forward pass uses a hard top-1 assignment; gradients pass through a straight-through softmax at a temperature that is annealed during training. Queries are mapped to the same codebook, and the selected address replaces the raw feature map in both the write and the read. The cost is one $N\times D$ similarity per token, small for $N$ in the hundreds. The codebook is frozen.
Complex ETFs, such as hyperoval or Tremain frames, are used through realification in $\mathbb{R}^{2D}$. The real inner product never exceeds the complex modulus, so the coherence bound carries over. We report results against the doubled real dimension so that state sizes stay honest.
The analytical work derives the expected worst-decile recall as a function of $(D, N, M)$ from collision probability plus $(M-1)\mu^2$ interference. We do this for exact ETFs and for general frames, characterizing the latter by the distribution of subset interference rather than its mean. We also check whether the delta-rule write used by DeltaNet-style and ATLAS memories changes the picture. Least-squares writes cancel interference among up to $D$ items but not beyond, so the packing should matter only in the overloaded regime $M>D$. We test that prediction.
Finally, we tabulate which $(D,N)$ pairs near common head dimensions admit exact ETFs from the families above. For each such "on-grid" pair we pick a nearby "off-grid" pair whose best numerical packing sits measurably above the Welch bound. This lets construction difficulty enter as a measured effect rather than a footnote.
Baselines
The naive reference is the sliding-window-only model with no global stream. It fixes the floor for far-context recall, and its failure on distant facts is documented in Simple linear attention language models balance the recall-throughput tradeoff. Full softmax attention at the same width serves as the upper reference.
The strongest existing competitor is BASED itself: Taylor linear attention plus a 64-token window, with learned feature maps. It is the hybrid that most directly targets the recall bottleneck we study, and its kernels are public in design. The number reported on Lacuna is that BASED at 1.3B parameters "outperformed Mamba by 10.36 accuracy points on recall-intensive tasks" (Simple linear attention language models balance the recall-throughput tradeoff). No per-configuration MQAR accuracy for BASED was found on Lacuna, so we reproduce it ourselves.
For fairness, every model shares the same token budget, data order, window size and global-state size. The global state is $D\times d_v$ per head. Raw feature dimension is not equalized, because BASED's Taylor map expands features; we instead match the stored-state floats and report both accounting choices. Each model receives the same learning-rate sweep, which also follows up on the learning-rate brittleness reported in Revisiting Associative Recall in Modern Recurrent Models. Beating BASED means higher worst-decile MQAR recall at matched state size, without losing more than the noise margin in Pile perplexity.
Three ablations isolate the key ingredient, and all keep the same quantized-address machinery so that only the geometry changes:
- Random codebook. Gaussian unit vectors at the same $(N,D)$ test whether equiangularity matters beyond the average-case identity.
- Tight but not equiangular. A codebook formed as a union of $N/D$ orthonormal bases is tight, so its average crosstalk is identical, but its overlaps are either zero or large. This is the $p<2$ minimizer of Ehler and Okoudjou.
- Bigger orthonormal state. A codebook with $N=D$ has no crosstalk but maximal collisions.
Why not just…?
Why not just add a coherence penalty on the learned keys? This is the most serious alternative. An ETF view of Dropout regularization shows that explicit coherence penalties help, raising Fashion-MNIST accuracy from 88.36% to 90.89%. The same paper also reports that such optimization is non-convex, costly, and does not reach the Welch bound. Keys are also data-dependent, so the penalty constrains only the keys seen in a batch, not the full address set. We run a per-sequence Gram-coherence penalty on BASED keys as an extra baseline under the same budget. If it matches the fixed codebook on worst-decile recall, the codebook's contribution reduces to "lower coherence by any means," and we would report that as the finding.
Why not just use random addresses, which are nearly incoherent in high dimension? Our own identity says that random and tight frames have nearly the same average crosstalk, so on mean recall this alternative should tie. Asymptotic Frame Theory for Analog Coding reports that random frames have spread-out spectra on random subsets while ETFs do not. The random-codebook ablation tests whether that difference reaches worst-case recall. If it does not, the proposal fails, and the test is designed so that it can.
Why not just use a delta-rule or test-time-trained memory, which cancels interference directly? Least-squares writing removes crosstalk only while the number of stored items stays at or below the state dimension. Beyond that, some lossy superposition is unavoidable, and the address geometry should again set the error. We therefore cross the address choice with the write rule and expect the packing benefit to appear only for $M>D$. If delta-rule writes erase the benefit even in that regime, the contribution is limited to plain linear-attention hybrids.
Why not just widen the sliding window? A window recalls nothing beyond its span. Long Range Arena reports universal failure on the 16K Pathfinder-X task. A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention proves that block-sparse attention cannot solve 2-Sum. The window-only baseline quantifies this in our setting.
Proposed Experiments
The first experiment is synthetic MQAR, as used in Simple linear attention language models balance the recall-throughput tradeoff. We control the number of far-context pairs $M$ placed beyond the window and sweep $N$ at fixed $D$. The metrics are worst-decile recall and mean recall, measured against the predicted curves. The decisive check is that equiangular codebooks beat both random and union-of-bases codebooks on worst-decile recall while roughly matching them on mean recall. That pattern is the signature of the subset-uniformity mechanism. If equiangular codebooks also win on mean recall, the mechanism is something other than the one claimed. If they fail to win on worst-decile recall, the mechanism is absent.
A second check compares on-grid and off-grid sizes. Exact-ETF configurations should sit on the predicted curve, and nearby off-grid numerical packings should fall short in proportion to their excess coherence over the Welch bound. Seeing this gap would show that ETF construction difficulty has a measurable cost in this architecture.
Third, we pretrain 360M-parameter models on the Pile, the scale BASED reports. We evaluate perplexity, the recall-intensive suite BASED uses, and BABILong at lengths up to the training context and four times beyond. We report throughput against BASED to show that the codebook lookup adds no meaningful cost.
The proposal succeeds if worst-case far-context recall improves at equal state size and perplexity is unchanged. It fails cleanly if random addresses perform as well.
References
- Grassmannian frames with applications to coding and communication
- Simple linear attention language models balance the recall-throughput tradeoff
- Long-Short Transformer: Efficient Transformers for Language and Vision
- AllMem: A Memory-centric Recipe for Efficient Long-context Modeling
- ATLAS: Learning to Optimally Memorize the Context at Test Time
- Asymptotic Frame Theory for Analog Coding
- Robust dimension reduction, fusion frames, and Grassmannian packings
- Minimization of the probabilistic p-frame potential
- Packings in Real Projective Spaces
- Kirkman Equiangular Tight Frames and Codes
- Tremain equiangular tight frames
- Equiangular Tight Frames From Hyperovals
- Optimal Line Packings from Nonabelian Groups
- An ETF view of Dropout regularization
- Early Exit with Disentangled Representation and Equiangular Tight Frame
- Equiangular Basis Vectors
- Revisiting Associative Recall in Modern Recurrent Models
- Long Range Arena : A Benchmark for Efficient Transformers
- A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attention