Abstract
Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have become the preferred choice for adapting Large Language Models (LLMs) due to their significantly lower computational requirements. However, this paper presents a theoretical and empirical "reality check," demonstrating that PEFT is a strict subset of Full Fine-Tuning (FFT). We prove that PEFT operates on a low-dimensional manifold that fundamentally limits its representational capacity and robustness. Our findings show that while PEFT is effective for simple tasks, it hits a performance ceiling on complex reasoning and is significantly more vulnerable to adversarial perturbations than FFT.
Introduction
As the scale of LLMs continues to grow, updating every parameter through Full Fine-Tuning (FFT) has become prohibitively expensive. This has led to the widespread adoption of PEFT techniques, which update only a tiny fraction of the model's weights. While PEFT often matches FFT performance on standard benchmarks, a performance gap persists in complex domains such as mathematical reasoning and instruction following. This research moves beyond empirical observations to provide a mathematical framework for comparing the optimization landscapes and representational limits of PEFT versus FFT.
Theoretical Foundations: Manifold Theory
The core of our theoretical analysis lies in the proof that PEFT operates on a "measure-zero manifold" within the full parameter space. Mathematically, this implies that the set of solutions reachable by PEFT is a vanishingly small, restricted slice of the model's total potential.
Furthermore, we utilize Hessian matrices to analyze the curvature of the loss landscape. Our analysis reveals that PEFT tends to converge to "sharper" minima. In the context of deep learning, sharper minima are generally associated with poorer generalization and increased sensitivity to small changes in input data.
The Representational Ceiling: The $k/d$ Penalty
A significant theoretical contribution of this work is the derivation of complexity bounds for PEFT. We demonstrate a "Rule of Diminishing Marginal Benefit": as the rank or number of trainable parameters ($k$) increases relative to the total parameters ($d$), the marginal gains in performance rapidly decrease.
Specifically, Theorem 5 shows that the risk reduction for each additional training sample in PEFT is approximately $k/d$ times that of FFT. Given that $k$ is often less than 0.1% of $d$, PEFT is fundamentally less efficient at "absorbing" and learning from large, complex datasets. This suggests that PEFT primarily "repurposes" existing internal knowledge rather than learning truly novel knowledge structures.
Robustness and Generalization
The restricted nature of the PEFT subspace makes it inherently brittle. Because the model cannot "steer around" noise as effectively as FFT, it becomes highly susceptible to adversarial attacks. Our experiments on the AdvGLUE benchmark highlight this vulnerability.
As shown in the table above, for the RoBbase model, FFT maintains an average adversarial accuracy of 36.58%, while LoRA drops to 30.89%—a gap of 5.69%. This "robustness collapse" indicates that PEFT solutions are less stable when faced with out-of-distribution perturbations.
Empirical Evidence: Weight Distribution
The theoretical constraints of PEFT are visible in the distribution of weight updates ($\Delta W$). Visual analysis shows that PEFT updates are highly concentrated around zero, whereas FFT updates are more widely distributed.
This wider distribution in FFT allows the model to undergo more significant structural changes, which is likely necessary for mastering complex tasks like the GSM8k math benchmark, where FFT-based LLaMA2-7B outperformed LoRA by 7.31%.
Conclusion
While PEFT is an invaluable tool for resource-constrained environments, it is not a perfect substitute for Full Fine-Tuning. The theoretical upper bounds and empirical evidence presented here suggest that PEFT is a "leaky abstraction" that eventually hits a wall in complex or adversarial settings. For tasks requiring high robustness or the learning of entirely new knowledge domains, Full Fine-Tuning remains the gold standard. Future research should look beyond established PEFT methods toward hybrid strategies that can bridge this capacity gap.
Summary updated