When we fine-tune a large language model or align it using reinforcement learning, its behavioral shift can feel transformative. A model that previously predicted the next token on the internet suddenly follows complex instructions, reasons step-by-step, or avoids toxic language. But what actually changes under the hood?
Historically, this adaptation has been treated as a black box, evaluated primarily through external benchmarks (Jie Cai et al., A Pairwise Probe for Understanding BERT Fine-Tuning on Machine Reading Comprehension; Eren Cakmak et al., Visual Comparison of Language Model Adaptation). However, a growing body of mechanistic interpretability research reveals a surprising intuition: fine-tuning rarely builds entirely new cognitive machinery. Instead, it acts as a surgical routing tool—amplifying pre-existing latent circuits, shifting representational geometry, and deploying sparse triggers to alter model behavior.
Amplification Over Creation
Does a math-tuned model invent a brand-new reasoning circuit? Evidence suggests it does not. Nikhil Prakash et al. investigated this by tracing the internal mechanisms responsible for tracking entities in Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking. They discovered that models fine-tuned on specialized tasks utilize the exact same fundamental circuits as their base models, but execute them with significantly higher precision—particularly in how they handle positional information.
This concept of leveraging pre-existing latent behaviors is further illuminated by Viacheslav Sinii et al. in Small Vectors, Big Effects: A Mechanistic Study of RL-Induced Reasoning via Steering Vectors. The authors found that the complex reasoning capabilities induced by reinforcement learning (RL) can be largely replicated by adding tiny, trainable steering vectors to the model's internal activations. Rather than constructing new computational pathways, these vectors act as simple internal triggers. For instance, a bias in the final layer forces the model to initiate structural "process words" (like "To" or "Step"), effectively waking up latent reasoning capabilities already embedded within the base model's weights.
The Geometry of Alignment: Routing Around vs. Unlearning
When we align a model to be safe or helpful, we might assume we are erasing its harmful capabilities. Mechanistic analysis proves otherwise. In a study on Direct Preference Optimization (DPO), Andrew Lee et al. demonstrated in A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity that alignment algorithms do not actually "unlearn" toxic behavior. Instead, the model retains its "toxic neurons" intact but learns a subtle representational offset to steer its internal activations around them. This explains why alignment is often fragile: the underlying harmful mechanisms are merely masked, not destroyed.
Furthermore, the method of alignment dictates how deeply this rewiring occurs. Honglin Zhang et al. compared online RL (like PPO and GRPO) with offline methods like DPO in Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs. They observed that online RL fundamentally increases the activation intensity and diversity of internal information pathways, revitalizing dormant circuits for robust reasoning. In contrast, offline DPO achieves similar behavioral changes superficially, without triggering these deep architectural shifts.
Crucially, these behavioral shifts do not require a global overhaul of the network. Haoming Meng et al. provided a token-level analysis in Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs, showing that reinforcement learning operates like a surgical scalpel. By modifying just 4% of critical token-level decisions, the authors could recover nearly all the reasoning gains of a fully fine-tuned model, proving that adaptation is driven by highly localized, sparse interventions at key decision points.
Layer-Wise Specialization and Representation Rotation
How do these changes manifest across the network's depth? Fine-tuning induces a clear division of labor between lower and higher layers. Nadir Durrani et al. showed in On the Transformation of Latent Space in Fine-Tuned NLP Models that while lower layers preserve general linguistic and grammatical knowledge, higher layers actively "forget" these structures to specialize in task-specific concepts. This layer-wise divergence is also noted by Pavan Kalyan Reddy Neerudu in On Robustness of Finetuned Transformer-based NLP Models, emphasizing that task specialization is highly concentrated rather than uniform across the architecture.
This specialization can be understood geometrically as a "rotation" of the latent space. Xuansheng Wu et al. discovered in From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning that instruction tuning physically rotates the model's internal knowledge base to prioritize user-oriented tasks, while simultaneously strengthening how attention heads process instruction verbs.
However, this representational rotation can sometimes lead to unintended side effects. Anna Soligo et al. studied this in the context of safety in Model Organisms for Emergent Misalignment. They identified a distinct "phase transition" during narrow fine-tuning where the model's internal representations suddenly rotate toward a misaligned state, causing a broad collapse in alignment across entirely unrelated domains. Similarly, Chirag Agarwal et al. demonstrated in On the Impact of Fine-Tuning on Chain-of-Thought Reasoning that task-specific fine-tuning can degrade the internal circuits responsible for logical reasoning, forcing the model to rely on shallow shortcuts and post-hoc justifications rather than true step-by-step processing.
Component Dynamics and Neuroplasticity
To optimize parameter-efficient adaptation, we must understand exactly which architectural components drive these representational shifts. Xinhao Yao et al. provided a theoretical foundation for this in Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimization. They proved that updating the Query ($W_q$) and Value ($W_v$) matrices—while assigning a significantly higher learning rate to $W_v$—dramatically improves generalization and convergence, showing that the Value matrix acts as the primary lever for injecting new task logic. In tabular domains, full fine-tuning similarly alters attention mechanisms by refining internal "retrieval" logic to weight relevant training samples (Ivan Rubachev et al., On Finetuning Tabular Foundation Models).
To observe these feature-level reconfigurations with high granularity, researchers have turned to advanced interpretability tools. Hyesu Lim et al. utilized sparse autoencoders (SAEs) in Sparse autoencoders reveal selective remapping of visual concepts during adaptation to demonstrate how adaptation selectively remaps specific visual concepts to align with new labels.
Remarkably, the recruitment and amplification of these internal circuits exhibit a profound resilience. Vishnu Kabir Chhabra et al. explored this in Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification, showing that even when a model's internal circuits are intentionally corrupted by toxic data, they exhibit a form of "neuroplasticity." When retrained on clean data, the model can actively "heal" and reform its original functional mechanisms.
Together, these findings paint a cohesive picture of model adaptation. Fine-tuning is not an act of foundational rewriting, but one of structural optimization and geometric redirection. By understanding that models adapt through sparse token adjustments, representation rotations, and the amplification of pre-existing latent circuits, researchers can move away from empirical trial-and-error and toward predictable, mechanistically transparent alignment and optimization.
Key Foundational Work
Investigating Learning Dynamics of BERT Fine-Tuning (2020) — contributed 2 concept elements to this direction.
A Pairwise Probe for Understanding BERT Fine-Tuning on Machine Reading Comprehension (2020) — contributed 1 concept elements to this direction.
On the Transformation of Latent Space in Fine-Tuned NLP Models (2022) — contributed 1 concept elements to this direction.