TL;DR
This paper reveals that the internal "geometry" of large language models (LLMs) doesn't just smoothly improve during training. Instead, it undergoes distinct, non-monotonic phases during pre-training ("warmup," "entropy-seeking," "compression-seeking") and post-training (SFT, DPO, RLVR). These geometric shifts are crucial for understanding how LLMs learn memorization, generalization, and alignment with specific tasks.
What Problem Does This Paper Address?
While we can track loss curves and task performance of LLMs, understanding how their internal representations evolve and what drives emergent capabilities remains a challenge. The paper posits that the "geometry" of these high-dimensional representations—how information is structured and distributed—is key to this understanding. Specifically, it's unclear how this geometry changes during the extensive pre-training and subsequent fine-tuning stages, and how these changes relate to different learning behaviors like memorization versus generalization.
Methods
The core of the paper's methodology lies in spectral analysis of the feature covariance matrix of the last-layer, last-token representations. They use two key metrics derived from this:
- Effective Rank (RankMe): This metric, based on Von Neumann entropy, quantifies the effective dimensionality of the representation space. A higher RankMe suggests a more spread-out, higher-dimensional representation, while a lower RankMe indicates compression into fewer dimensions.
- Spectral Decay Rate (aReQ): This measures how quickly the eigenvalues of the covariance matrix decay. A slower decay (smaller aReQ) implies higher dimensionality, while a faster decay (larger aReQ) indicates a more compact, anisotropic geometry.
To quantify distributional memorization, they use the Spearman rank correlation between an LLM's n-gram predictions and those of a generalized n-gram language model trained on the same corpus.
They also employ a toy model to analytically study the dynamics of gradient descent under cross-entropy loss, specifically investigating the roles of information bottleneck and skewed data distributions.
Datasets
- Pre-training: The analysis uses checkpoints from the Pythia Suite (70M to 12B parameters) and the OLMo Framework (e.g., OLMo-7B). The pre-training data for these models is not explicitly detailed beyond mentioning the Pile dataset for Pythia and Dolma for OLMo.
- Post-training: They analyze Tülu-3.1 models (LLaMA-based) which have undergone Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning from Verifiable Rewards (RLVR).
- Evaluation: For spectral analysis during pretraining, they use sequences from the FineWeb dataset. For downstream task evaluation, they mention SciQ and TriviaQA.
Key Insights and Ideas
- LLM training is not monotonic: Contrary to a smooth improvement in loss, the internal geometry of LLMs undergoes distinct, non-monotonic phases during both pre-training and post-training.
- "Entropy-seeking" vs. "Compression-seeking": These are the two dominant geometric dynamics. "Entropy-seeking" involves manifold expansion and increased dimensionality, correlating with memorization. "Compression-seeking" involves anisotropic consolidation, reducing dimensionality along dominant axes, correlating with generalization and learning long-range dependencies.
- Pre-training phases are consistent: Across different model families (Pythia, OLMo) and scales, pre-training consistently shows a "warmup" phase (rapid compression), followed by "entropy-seeking" (expansion/memorization), and then "compression-seeking" (consolidation/generalization).
- Post-training refines geometry: SFT and DPO tend to induce "entropy-seeking" dynamics, aiding instruction following but potentially hurting out-of-distribution robustness. RLVR, conversely, promotes "compression-seeking," consolidating reward-aligned behaviors and reducing exploration.
- Geometric phases explain emergent behaviors: The "entropy-seeking" phase is linked to memorizing short-context statistics (n-grams), while the "compression-seeking" phase is crucial for learning long-context dependencies and generalization, explaining phenomena like grokking.
- Toy model confirms dynamics: A simplified analytical model demonstrates that skewed data distributions and information bottlenecks are critical for replicating these multiphase geometric dynamics through gradient descent.
Key Results
- Consistent Pre-training Phases: Spectral analysis (RankMe and aReQ) on OLMo-2 and Pythia models consistently revealed three distinct pre-training phases: "warmup," "entropy-seeking," and "compression-seeking."
- Memorization Peaks in "Entropy-Seeking": The distributional memorization metric peaked during the "entropy-seeking" phase, indicating its importance for learning short-context statistics.
- Generalization Benefits from "Compression-Seeking": Factual question-answering tasks (TriviaQA) showed improvement primarily during the "compression-seeking" phase, highlighting its role in learning long-context dependencies.
- Post-training Dynamics: SFT and DPO showed monotonic increases in RankMe ("entropy-seeking"), while RLVR showed monotonic decreases ("compression-seeking").
- Trade-offs in Post-Training: SFT's "entropy-seeking" improved in-distribution performance but could reduce out-of-distribution robustness. RLVR's "compression-seeking" improved performance on specific tasks (e.g., pass@16) but potentially limited broader exploration (pass@256).
Limitations and Future Work
- Limitations: The paper acknowledges that while their toy model provides insights, it's a simplification of real LLMs. The exact mechanisms by which these geometric shifts translate to specific capabilities are still an active area of research. The paper also notes that SFT can reduce out-of-distribution robustness.
- Future Work: The authors suggest that spectral measures of representation geometry are a powerful tool for understanding LLM learning dynamics. Future work could involve exploring these geometric shifts in more detail across various architectures, training objectives, and datasets, and further investigating how these geometric properties directly influence specific emergent abilities and potential failure modes. They also hint at an iterative "exploration-consolidation cycle" in pre-training and how post-training recipes induce characteristic shifts.
Summary updated