When we train, fine-tune, or prompt a Large Language Model (LLM), what actually happens inside its hidden layers? Instead of viewing a model as a static database of facts or an opaque black box of weights, a growing body of research treats the model's internal activations—specifically its residual stream—as a dynamic, high-dimensional geometric space.
The core intuition of this research direction is that learning, adapting, and reasoning are fundamentally geometric transformations. Rather than just accumulating isolated data points, models build, rotate, and align structured coordinate systems to process information.
Pre-Training and the Crystallization of Structure
During the massive pre-training phase, a model's representation space undergoes dramatic structural shifts. Rather than smoothly improving, the geometry evolves through distinct, non-monotonic phases. As traced by Melody Zixuan Li et al. in Tracing the representation geometry of language models from pretraining to post-training, the representation space moves through "warmup," "entropy-seeking," and "compression-seeking" phases. These phases map directly to how the model balances memorization and generalization.
As these phases progress, the high-dimensional highway of the residual stream begins to stabilize. Jacek Karwowski et al. demonstrate in Characterizing stable regions in the residual stream of LLMs that the model organizes its internal representations into large, semantically consistent neighborhoods that "crystallize" over time.
Within these neighborhoods, different types of information are encoded along different geometric dimensions. For instance, Jin Hwa Lee et al. found in GEOMETRIC SIGNATURES OF COMPOSITIONALITY ACROSS A LANGUAGE MODEL’S LIFETIME that linear dimensions tend to capture superficial word properties, while nonlinear dimensions gradually emerge over training to handle complex semantic compositionality.
Remarkably, this geometric trajectory is not random or highly sensitive to minor architectural tweaks. Christopher Wolfram et al. showed in Layers at Similar Depths Generate Similar Activations Across LLM Architectures that layers at similar depths across entirely different LLM architectures generate highly similar activations. This suggests a universal developmental path for how language models transform and structure information geometrically.
Fine-Tuning as Knowledge Reorganization
What happens when we take a pre-trained model and fine-tune it for a specific task or instruct it to follow user prompts? A central question has been whether fine-tuning injects entirely new capabilities or simply uncovers existing ones. Geometric analysis strongly supports the latter: fine-tuning is primarily a process of reorganizing and re-aligning the existing representation space.
In What Happens To BERT Embeddings During Fine-tuning?, Amil Merchant et al. reveal that fine-tuning is a highly conservative process. It leaves the foundational lower layers largely intact—avoiding the catastrophic forgetting of general linguistic knowledge—and primarily modifies the geometry of the top layers to suit the target task.
When this adaptation happens at scale—such as during instruction tuning—the internal geometry undergoes a coordinated shift. Xuansheng Wu et al. explore this in From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning, demonstrating that instruction tuning doesn't necessarily build new linguistic structures from scratch. Instead, it effectively "rotates" the model's internal knowledge base, shifting the representation space to prioritize user-oriented tasks and strengthening how attention heads process instruction verbs.
However, this geometric shifting can introduce trade-offs. Aliyah R. Hsu et al. show in Diagnosing Transformers: Illuminating Feature Spaces for Clinical Decision-Making that while domain-specific fine-tuning optimizes feature spaces for targeted tasks, it can distort the original representation space, making models prone to overfitting on imbalanced data. To understand whether these changes stem from entirely new functional components or just altered communication pathways, Xu Wang et al. leverage circuit analysis in Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis to map out how internal representations structurally reorganize during learning.
Dynamic and Sequential Geometric Shifts
This geometric perspective also explains how models process information dynamically during inference and how they react to discrete, sequential updates.
Consider In-Context Learning (ICL), where a model learns from a few examples provided directly in the prompt without any weight updates. Haolin Yang et al. explain this in Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning as a two-stage geometric process operating entirely within the hidden states during a single forward pass. Early layers geometrically separate hidden states based on their semantic labels, while later layers align these states with the correct output vectors. ICL succeeds by dynamically fixing alignment issues in the representation space on the fly.
Conversely, when we force explicit parameter changes through sequential model editing, the underlying geometry can resist or degrade if pushed too far. Qi Li et al. study this in Can We Continually Edit Language Models? On the Knowledge Attenuation in Sequential Model Editing, showing that sequential updates can cause "Knowledge Attenuation" because the physical rearrangement of internal representations decays over multiple discrete edits.
To make these complex, high-dimensional shifts interpretable, researchers have developed frameworks to translate geometric states back into human-comprehensible forms. For example, Johannes Schneider 0002 et al. introduced ClaDec in Explaining Neural Networks by Decoding Layer Activations, a method that decodes internal layer activations into visual reconstructions. This allows us to see exactly what features are preserved or discarded as representations evolve toward a final decision.
Why the Geometric View Works
By viewing neural network training and adaptation through the lens of representation geometry, the black box becomes a structured, evolving landscape. Pre-training builds and crystallizes the foundational semantic neighborhoods; fine-tuning and instruction tuning rotate and optimize these axes for downstream tasks; and inference dynamically steers hidden states along these geometric pathways to generate final outputs. Understanding these spatial dynamics allows researchers to diagnose model failures, prevent overfitting, and design more efficient alignment strategies.
Key Foundational Work
What Happens To BERT Embeddings During Fine-tuning? (2020) — contributed 1 concept elements to this direction.
Explaining Neural Networks by Decoding Layer Activations (2021) — contributed 1 concept elements to this direction.