This paper offers a fascinating look under the hood of Large Language Models (LLMs). By treating LLMs as model-agnostic probabilistic sequences, the authors uncover a hidden "low-rank" structure that governs how these models predict text.
TL;DR
The authors demonstrate that the log-probabilities (logits) of modern LLMs across various prompts and responses form a low-rank matrix. They leverage this structure to develop LINGEN, a method that can generate coherent text for a target prompt by linearly combining the model's outputs from completely unrelated, or even nonsensical, prompts.
What Problem Does This Paper Address?
Despite their massive success, LLMs remain mathematical "black boxes." We lack simple, universal abstractions that explain how they represent information over long sequences. While researchers have previously identified a "softmax bottleneck" (a low-rank property when predicting a single next token), it wasn't clear if this low-dimensionality persisted over longer horizons.
The authors set out to find a model-agnostic framework—one that doesn't care if the model is a Transformer, a State Space Model (SSM), or something else—to describe the inherent low-dimensional structure of language as captured by these models.
Methods
The core of their approach is the Extended Logit Matrix.
- The Matrix: Imagine a giant table where rows are "histories" (prompts) and columns are "futures" (possible completions). Each entry is the log-probability the model assigns to that future given that history.
- Empirical Rank Analysis: The authors used Singular Value Decomposition (SVD) on sub-matrices of real LLMs to see how quickly the "importance" of different dimensions drops off.
- LINGEN (Linear Generation): This is the paper's most striking application. If the logit matrix is low-rank, any "target" prompt can be approximated as a linear combination of other "basis" prompts. LINGEN generates text for a target prompt by querying the model only on these basis prompts and combining the results.
- Theoretical Modeling: They connect this low-rank property to a specific architecture called a Time-varying Input Switched Affine Network (ISAN), proving that a model has a low-rank logit matrix if and only if it can be represented as an ISAN.
Datasets
The researchers evaluated several open-source models, including OLMo (1b and 7b), Llama-1b, Gemma-1b, and Mamba-1.4b. To build their logit matrices, they sampled sequences from diverse datasets:
- Standard: Wiki, Arxiv, Starcoder, Math, and C4.
- Nonsense: They created "nonsense" versions of these datasets by randomly permuting tokens to test if the low-rank structure was tied to semantic meaning or was a more fundamental property of the model's "world view."
Key Insights and Ideas
- The Power Law of Language: The singular values of the logit matrices decay according to a power law ($i^{-\alpha}$). Crucially, $\alpha$ is typically greater than 0.5, which is the mathematical "phase transition" point where a matrix can be effectively approximated by a low-rank version.
- Structure is Learned, Not Born: The low-rank structure is absent in untrained models (Step 0). It emerges rapidly during the early stages of pre-training and continues to refine as the model sees more data.
- Semantic Invariance: Surprisingly, the linear relationships between prompts remain consistent even when evaluated against "nonsense" futures. This suggests that the model's internal representation of a prompt's "meaning" is robust and independent of the specific future it is predicting.
- Equivalence to SSMs: The authors show that their low-rank abstraction can represent State Space Models (SSMs), bridging the gap between theoretical low-rank models and practically successful architectures.
Key Results
- LINGEN Performance: LINGEN successfully generates coherent text. In terms of KL divergence (a measure of how close the generated distribution is to the "true" model), LINGEN significantly outperformed strong baselines, including models with restricted context windows and intermediate training checkpoints.
- Generation from Nonsense: LINGEN worked even when the "basis" prompts were random strings of tokens. This proves that the model's response to gibberish still contains enough structural information to reconstruct its response to a coherent target prompt.
- Provable Learning: On the theoretical side, the authors provide an algorithm that can "steal" or learn a low-rank model using only logit queries, offering new insights into model security and interpretability.
Limitations and Future Work
- The "Target" Query: Currently, to find the linear coefficients for LINGEN, the authors still need to query the model on the target prompt at least once. They suggest that "model transferability"—using a smaller, cheaper model to find the coefficients for a larger one—is a promising next step.
- Safety and Jailbreaking: The ability to generate text via unrelated prompts has serious implications for AI safety. It suggests a potential path to bypass "prompt filters" designed to block harmful content, as the harmful intent is never explicitly sent to the model in a single query.
- Approximate vs. Exact: While the theory handles exact low rank, real models are approximately low-rank. Bridging this gap more rigorously is an open theoretical challenge.
By showing that the "messy" output of an LLM is actually governed by clean, linear algebraic rules, this paper opens up new ways to steer, compress, and understand the next generation of AI.
Summary updated