Sequences of Logits Reveal the Low Rank Structure of Language Models

Noah Golowich, Allen Liu, Abhishek Shetty

TL;DR

This paper demonstrates that the "extended logit matrices" of modern large language models (LLMs)—which map prompts to multi-token future completions—exhibit a striking, architecture-agnostic low-rank structure. Remarkably, this low-rank property allows researchers to generate coherent text responses to a target prompt by using a linear combination of the model's outputs on completely unrelated, or even nonsensical, prompts. The authors formalize this finding theoretically by proving an equivalence between low logit rank and expressibility via a simple recurrent abstraction called a time-varying Input Switched Affine Network (ISAN).

What Problem Does This Paper Address?

While Large Language Models (LLMs) excel at processing and generating text, understanding their underlying mathematical structure remains a major challenge. Previous work identified the "softmax bottleneck," showing that the matrix of next-token logits is constrained by the model's hidden dimension. However, this only explains a single token step into the future.

This paper tackles a much broader question: Does the low-dimensional structure of language models persist over longer sequences of tokens? Furthermore, can we analyze this structure in an architecture-agnostic way, treating the LLM purely as a sequential probabilistic model rather than diving into specific transformer weights or attention maps?

Methods

The authors introduce the concept of the extended logit matrix. In this matrix, rows correspond to "histories" (prompts) and columns correspond to "futures" (possible multi-token completions) multiplied by the next-token vocabulary. The entries are the mean-centered log-probabilities of the future given the history.

Because the full matrix is exponentially large, they evaluate submatrices using two main methods:

  1. Singular Value Decay: Checking if the singular values follow a power law ($C \cdot i^{-\alpha}$). A phase transition occurs at $\alpha = 1/2$; if $\alpha > 1/2$, the matrix can be well-approximated by a constant-rank matrix.
  2. Average KL Divergence: Measuring how well a low-rank approximation preserves the true next-token probability distributions.

They leverage this structure to create LINGEN (Linear Generation), an algorithm that expresses a target prompt as a linear combination of unrelated prompts. By querying the model only on these unrelated prompts and combining their logits, LINGEN can generate text for the target prompt.

On the theoretical side, they prove that a model has an exact low logit rank if and only if it can be perfectly represented as a time-varying Input Switched Affine Network (ISAN)—a simple recurrent architecture with linear updates and a softmax output.

Datasets

The empirical evaluations were performed using text from several diverse datasets to construct the history and future sets:

  • The wiki, arxiv, and starcoder subsets of the olmo-mix-1124 dataset.
  • The math subset of the dolmino-mix-1124 dataset.
  • The c4 dataset.

Key Insights and Ideas

  • Universal Power-Law Decay: Across different models (transformers like OLMo, Llama, Gemma, and state-space models like Mamba), the singular values of the extended logit matrix decay according to a highly consistent power law with an exponent $\alpha > 1/2$. This means the low-rank structure is a fundamental property of trained language models, not an artifact of a specific architecture.
  • Emergence During Training: This low-rank structure is not present at initialization (Step 0). It emerges rapidly during the early stages of pre-training and stabilizes, suggesting that learning the low-dimensional structure of language is core to the pre-training process.
  • Linear Relationships Transfer Globally: The linear relationships between prompts (the row kernel of the matrix) remain preserved even if you completely replace natural language futures with random, nonsensical token sequences.
  • Generation via Nonsense: Because the subspace structure transfers so robustly, you can generate coherent text for a target prompt by querying the model only on random, permuted nonsense prompts. This has massive implications for AI safety, as it could allow attackers to bypass input prompt filters entirely.

Key Results

  • Power-Law Exponents: For models like OLMo-7b, Gemma-1b, Llama-1b, and Mamba-1.4b, the power-law exponent $\alpha$ consistently ranges between $0.53$ and $0.58$, safely above the $0.5$ phase transition threshold. At step 0 of training, however, $\alpha$ drops to $\approx 0.37$, indicating a lack of low-rank structure.
Singular value decay for OLMo-7b showing power-law behavior
  • LINGEN Performance: When generating text using out-of-distribution, randomized "nonsense" prompts, LINGEN achieved a significantly lower total KL divergence (4.95) compared to strong baselines like restricting the context window (17.56) or using a single-token logit matrix variant (14.41).
LINGEN generation performance and low-rank approximation error compared to baselines
  • Subspace Overlap: Computing the principal angles between the column spaces of logit matrices from different models revealed significant overlap, proving that different LLMs learn highly compatible low-dimensional representations of text.

Limitations and Future Work

  • Loose Theoretical Bounds: While the authors provide provable generalization bounds for LINGEN, they note that these bounds are "lossy" and not yet tight enough to directly guide practical engineering without empirical tuning.
  • Black-Box Mechanics: The paper clearly documents that low-rank structure emerges early in training, but why and how the optimization dynamics of transformers force this specific structure remains an open question.
  • Safety Exploits: The ability to generate text via nonsense prompts suggests a novel vector for jailbreaking LLMs. Future work needs to explore how to defend against these types of coordinate/linear combination attacks that circumvent standard input filters.

Summary updated

Abstract

A major problem in the study of large language models is to understand their inherent low-dimensional structure. We introduce an approach to study the low-dimensional structure of language models at a model-agnostic level: as sequential probabilistic models. We first empirically demonstrate that a wide range of modern language models exhibit low-rank structure: in particular, matrices built from the model's logits for varying sets of prompts and responses have low approximate rank. We then show that this low-rank structure can be leveraged for generation -- in particular, we can generate a response to a target prompt using a linear combination of the model's outputs on unrelated, or even nonsensical prompts. On the theoretical front, we observe that studying the approximate rank of language models in the sense discussed above yields a simple universal abstraction whose theoretical predictions parallel our experiments. We then analyze the representation power of the abstraction and give provable learning guarantees.
BibTeX
@inproceedings{golowich2026sequences,
  author = {Noah Golowich and Allen Liu and Abhishek Shetty},
  title = {{Sequences of Logits Reveal the Low Rank Structure of Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=gdZ6J5hZzF},
}
Figure