Tracing the representation geometry of language models from pretraining to post-training

Melody Zixuan Li, Kumar Krishna Agrawal, Arna Ghosh, Komal Kumar Teru, Guillaume Lajoie, Blake Aaron Richards

TL;DR

This paper reveals that the internal "geometry" of large language models (LLMs) doesn't just smoothly improve during training. Instead, it undergoes distinct, non-monotonic phases during pre-training ("warmup," "entropy-seeking," "compression-seeking") and post-training (SFT, DPO, RLVR). These geometric shifts are crucial for understanding how LLMs learn memorization, generalization, and alignment with specific tasks.

What Problem Does This Paper Address?

While we can track loss curves and task performance of LLMs, understanding how their internal representations evolve and what drives emergent capabilities remains a challenge. The paper posits that the "geometry" of these high-dimensional representations—how information is structured and distributed—is key to this understanding. Specifically, it's unclear how this geometry changes during the extensive pre-training and subsequent fine-tuning stages, and how these changes relate to different learning behaviors like memorization versus generalization.

Methods

The core of the paper's methodology lies in spectral analysis of the feature covariance matrix of the last-layer, last-token representations. They use two key metrics derived from this:

  • Effective Rank (RankMe): This metric, based on Von Neumann entropy, quantifies the effective dimensionality of the representation space. A higher RankMe suggests a more spread-out, higher-dimensional representation, while a lower RankMe indicates compression into fewer dimensions.
  • Spectral Decay Rate (aReQ): This measures how quickly the eigenvalues of the covariance matrix decay. A slower decay (smaller aReQ) implies higher dimensionality, while a faster decay (larger aReQ) indicates a more compact, anisotropic geometry.

To quantify distributional memorization, they use the Spearman rank correlation between an LLM's n-gram predictions and those of a generalized n-gram language model trained on the same corpus.

They also employ a toy model to analytically study the dynamics of gradient descent under cross-entropy loss, specifically investigating the roles of information bottleneck and skewed data distributions.

Datasets

  • Pre-training: The analysis uses checkpoints from the Pythia Suite (70M to 12B parameters) and the OLMo Framework (e.g., OLMo-7B). The pre-training data for these models is not explicitly detailed beyond mentioning the Pile dataset for Pythia and Dolma for OLMo.
  • Post-training: They analyze Tülu-3.1 models (LLaMA-based) which have undergone Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning from Verifiable Rewards (RLVR).
  • Evaluation: For spectral analysis during pretraining, they use sequences from the FineWeb dataset. For downstream task evaluation, they mention SciQ and TriviaQA.

Key Insights and Ideas

  • LLM training is not monotonic: Contrary to a smooth improvement in loss, the internal geometry of LLMs undergoes distinct, non-monotonic phases during both pre-training and post-training.
  • "Entropy-seeking" vs. "Compression-seeking": These are the two dominant geometric dynamics. "Entropy-seeking" involves manifold expansion and increased dimensionality, correlating with memorization. "Compression-seeking" involves anisotropic consolidation, reducing dimensionality along dominant axes, correlating with generalization and learning long-range dependencies.
  • Pre-training phases are consistent: Across different model families (Pythia, OLMo) and scales, pre-training consistently shows a "warmup" phase (rapid compression), followed by "entropy-seeking" (expansion/memorization), and then "compression-seeking" (consolidation/generalization).
  • Post-training refines geometry: SFT and DPO tend to induce "entropy-seeking" dynamics, aiding instruction following but potentially hurting out-of-distribution robustness. RLVR, conversely, promotes "compression-seeking," consolidating reward-aligned behaviors and reducing exploration.
  • Geometric phases explain emergent behaviors: The "entropy-seeking" phase is linked to memorizing short-context statistics (n-grams), while the "compression-seeking" phase is crucial for learning long-context dependencies and generalization, explaining phenomena like grokking.
  • Toy model confirms dynamics: A simplified analytical model demonstrates that skewed data distributions and information bottlenecks are critical for replicating these multiphase geometric dynamics through gradient descent.

Key Results

  • Consistent Pre-training Phases: Spectral analysis (RankMe and aReQ) on OLMo-2 and Pythia models consistently revealed three distinct pre-training phases: "warmup," "entropy-seeking," and "compression-seeking."
  • Memorization Peaks in "Entropy-Seeking": The distributional memorization metric peaked during the "entropy-seeking" phase, indicating its importance for learning short-context statistics.
  • Generalization Benefits from "Compression-Seeking": Factual question-answering tasks (TriviaQA) showed improvement primarily during the "compression-seeking" phase, highlighting its role in learning long-context dependencies.
  • Post-training Dynamics: SFT and DPO showed monotonic increases in RankMe ("entropy-seeking"), while RLVR showed monotonic decreases ("compression-seeking").
  • Trade-offs in Post-Training: SFT's "entropy-seeking" improved in-distribution performance but could reduce out-of-distribution robustness. RLVR's "compression-seeking" improved performance on specific tasks (e.g., pass@16) but potentially limited broader exploration (pass@256).

Limitations and Future Work

  • Limitations: The paper acknowledges that while their toy model provides insights, it's a simplification of real LLMs. The exact mechanisms by which these geometric shifts translate to specific capabilities are still an active area of research. The paper also notes that SFT can reduce out-of-distribution robustness.
  • Future Work: The authors suggest that spectral measures of representation geometry are a powerful tool for understanding LLM learning dynamics. Future work could involve exploring these geometric shifts in more detail across various architectures, training objectives, and datasets, and further investigating how these geometric properties directly influence specific emergent abilities and potential failure modes. They also hint at an iterative "exploration-consolidation cycle" in pre-training and how post-training recipes induce characteristic shifts.

Summary updated

Abstract

The geometry of representations in a neural network can significantly impact downstream generalization. It is unknown how representation geometry changes in large language models (LLMs) over pretraining and post-training. Here, we characterize the evolving geometry of LLM representations using spectral methods (effective rank and eigenspectrum decay). With the OLMo and Pythia model families we uncover a consistent non-monotonic sequence of three distinct geometric phases in pretraining. An initial \warmup phase sees rapid representational compression. This is followed by an "entropy-seeking" phase, characterized by expansion of the representation manifold's effective dimensionality, which correlates with an increase in memorization. Subsequently, a "compression seeking" phase imposes anisotropic consolidation, selectively preserving variance along dominant eigendirections while contracting others, correlating with improved downstream task performance. We link the emergence of these phases to the fundamental interplay of cross-entropy optimization, information bottleneck, and skewed data distribution. Additionally, we find that in post-training the representation geometry is further transformed: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) correlate with another "entropy-seeking" dynamic to integrate specific instructional or preferential data, reducing out-of-distribution robustness. Conversely, Reinforcement Learning with Verifiable Rewards (RLVR) often exhibits a "compression seeking" dynamic, consolidating reward-aligned behaviors and reducing the entropy in its output distribution. This work establishes the utility of spectral measures of representation geometry for understanding the multiphase learning dynamics within LLMs.
BibTeX
@inproceedings{li2025tracing,
  author = {Melody Zixuan Li and Kumar Krishna Agrawal and Arna Ghosh and Komal Kumar Teru and Guillaume Lajoie and Blake Aaron Richards},
  title = {{Tracing the representation geometry of language models from pretraining to post-training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=FDruZlKWUb},
}
Figure