Geometric Signatures of Compositionality Across a Language Model's Lifetime

Jin Hwa Lee, Thomas Jiralerspong, Lei Yu, Yoshua Bengio, Emily Cheng

TL;DR

This paper reveals a fascinating "form-meaning dichotomy" in how language models (LMs) geometrically organize data: linear dimensionality captures the superficial structure (form) of text, while nonlinear intrinsic dimension tracks true semantic compositionality (meaning). Furthermore, tracking these metrics over training reveals a sharp geometric phase transition at around 1,000 steps, precisely marking the sudden onset of the model's linguistic competence.

What Problem Does This Paper Address?

Human language is incredibly complex yet inherently simple. Thanks to compositionality—the idea that the meaning of an expression is built from its parts using structured rules—a finite vocabulary and a few syntactic rules can generate an infinite variety of sentences.

According to the manifold hypothesis, high-dimensional data (like text embeddings) actually lies on a much lower-dimensional, nonlinear manifold. While we know language models compress text into lower dimensions, a fundamental question remains: Does the geometric complexity of an LM's internal representations actually reflect the compositionality of its inputs?

Additionally, the paper investigates how this geometric structure develops over time during pre-training and whether different types of dimensionality (linear vs. nonlinear) encode different aspects of language.

Methods

The authors employ two distinct lenses to measure representational dimensionality within the Transformer's residual stream (specifically looking at the last token representation):

  1. Nonlinear Intrinsic Dimension ($I_d$): Measured using the TwoNN estimator, which calculates the minimal number of degrees of freedom needed to describe the data manifold without losing information based on local neighbor distances.
  2. Linear Effective Dimension ($d$): Measured using standard Principal Component Analysis (PCA) with a 99% variance cutoff, capturing the global linear subspace the representations occupy.

To quantify the superficial complexity of the text datasets, they approximate Kolmogorov Complexity (KC) using the file size of the dataset compressed via gzip.

Datasets

  • Controlled Synthetic Grammar: To surgically manipulate compositionality, the authors designed a synthetic grammar generating 17-word sentences across 12 semantic categories (e.g., [quality] [nationality] [job] [action]...). They controlled compositionality by coupling $k$ contiguous word positions (sampling $k$-grams together), reducing the dataset's true degrees of freedom as $k$ increases. They also created a shuffled control dataset (destroying syntax/meaning while preserving word frequencies) to isolate superficial complexity from semantic complexity.
  • Naturalistic Language: To ensure findings generalize beyond synthetic data, key experiments were replicated using samples from The Pile.

Key Insights and Ideas

  • The Form-Meaning Dichotomy: Linear dimension ($d$) and nonlinear intrinsic dimension ($I_d$) behave completely differently. Linear dimension tracks the surface-level variation or form of the text. Shuffling words actually increases linear dimension because it increases superficial randomness. Conversely, nonlinear intrinsic dimension tracks meaning; shuffling words causes $I_d$ to collapse because the semantic compositional structure is destroyed.
  • Global Expansion vs. Local Compression: As models scale up in hidden dimension ($D$), their linear dimension expands proportionally ($d \propto D$). However, their nonlinear intrinsic dimension ($I_d$) remains robustly low and stable ($O(10)$). This implies that larger LMs use their vast capacity to spread representations out globally, but locally constrain them to a tight, low-dimensional manifold that captures the true underlying degrees of freedom of the language.
Linear dimension scales with model size, while nonlinear intrinsic dimension remains robustly low.
  • The Geometric Phase Transition: During training, LMs don't learn smoothly; they undergo a dramatic phase transition at around $10^3$ steps. At this exact moment, $I_d$ drops sharply and redistributes across layers, coinciding perfectly with the sudden emergence of downstream linguistic and reasoning capabilities.

Key Results

  • Superficial Complexity Tracking: Linear dimension ($d$) is almost perfectly correlated with the gzip-estimated Kolmogorov complexity across nearly all model families (Spearman $\rho \approx 0.9$ to $1.0$), proving it acts as a proxy for surface form.
  • Shuffling Collapse: For fully trained models, breaking sentence structure via shuffling causes a massive divergence: $I_d$ collapses to a narrow, low range across layers, while linear $d$ shoots upward compared to coherent text.
Divergence of linear and nonlinear dimensions across layers for coherent vs. shuffled text.
  • Linguistic Competence Alignment: The sudden shift in $I_d$ at $10^3$ steps aligns with rapid performance jumps in zero-shot benchmarks like SciQ, PIQA, and WinoGrande, showing that geometric complexity directly mirrors functional capability.
The phase transition in intrinsic dimension tracks the onset of benchmark task performance.

Limitations and Future Work

  • Syntactic Variety: The synthetic datasets used are restricted to a few specific, non-recursive syntactic structures. Future work should explore how more complex linguistic phenomena, like recursive embedding, impact these geometric signatures.
  • Causal Mechanics: While the paper establishes a strong correlation between representation geometry and compositionality, it does not definitively prove how these linear and nonlinear features causally drive predictive coding. Causal probing and feature attribution are needed.
  • Model Scale Constraints: Due to compute limitations, the training dynamics analysis was bounded to models up to 6.9B parameters and static analysis up to 8B parameters. Verifying these geometric properties in frontier-scale LLMs remains an open avenue.

Summary updated

Abstract

By virtue of linguistic compositionality, few syntactic rules and a finite lexicon can generate an unbounded number of sentences. That is, language, though seemingly high-dimensional, can be explained using relatively few degrees of freedom. An open question is whether contemporary language models (LMs) reflect the intrinsic simplicity of language that is enabled by compositionality. We take a geometric view of this problem by relating the degree of compositionality in a dataset to the intrinsic dimension (ID) of its representations under an LM, a measure of feature complexity. We find not only that the degree of dataset compositionality is reflected in representations' ID, but that the relationship between compositionality and geometric complexity arises due to learned linguistic features over training. Finally, our analyses reveal a striking contrast between nonlinear and linear dimensionality, showing they respectively encode semantic and superficial aspects of linguistic composition.
BibTeX
@inproceedings{lee2025geometric,
  author = {Jin Hwa Lee and Thomas Jiralerspong and Lei Yu and Yoshua Bengio and Emily Cheng},
  title = {{Geometric Signatures of Compositionality Across a Language Model's Lifetime}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure