Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning

Yongkang Liu, Xingle Xu, Ercong Nie, Zijing Wang, Shi Feng, Daling Wang, Qian Li, Hinrich SchützearXiv 2025 · OpenReview

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have become the preferred choice for adapting Large Language Models (LLMs) due to their significantly lower computational requirements. However, this paper presents a theoretical and empirical "reality check," demonstrating that PEFT is a strict subset of Full Fine-Tuning (FFT). We prove that PEFT operates on a low-dimensional manifold that fundamentally limits its representational capacity and robustness. Our findings show that while PEFT is effective for simple tasks, it hits a performance ceiling on complex reasoning and is significantly more vulnerable to adversarial perturbations than FFT.

Introduction

As the scale of LLMs continues to grow, updating every parameter through Full Fine-Tuning (FFT) has become prohibitively expensive. This has led to the widespread adoption of PEFT techniques, which update only a tiny fraction of the model's weights. While PEFT often matches FFT performance on standard benchmarks, a performance gap persists in complex domains such as mathematical reasoning and instruction following. This research moves beyond empirical observations to provide a mathematical framework for comparing the optimization landscapes and representational limits of PEFT versus FFT.

Theoretical Foundations: Manifold Theory

The core of our theoretical analysis lies in the proof that PEFT operates on a "measure-zero manifold" within the full parameter space. Mathematically, this implies that the set of solutions reachable by PEFT is a vanishingly small, restricted slice of the model's total potential.

Furthermore, we utilize Hessian matrices to analyze the curvature of the loss landscape. Our analysis reveals that PEFT tends to converge to "sharper" minima. In the context of deep learning, sharper minima are generally associated with poorer generalization and increased sensitivity to small changes in input data.

The Representational Ceiling: The $k/d$ Penalty

A significant theoretical contribution of this work is the derivation of complexity bounds for PEFT. We demonstrate a "Rule of Diminishing Marginal Benefit": as the rank or number of trainable parameters ($k$) increases relative to the total parameters ($d$), the marginal gains in performance rapidly decrease.

Specifically, Theorem 5 shows that the risk reduction for each additional training sample in PEFT is approximately $k/d$ times that of FFT. Given that $k$ is often less than 0.1% of $d$, PEFT is fundamentally less efficient at "absorbing" and learning from large, complex datasets. This suggests that PEFT primarily "repurposes" existing internal knowledge rather than learning truly novel knowledge structures.

Robustness and Generalization

The restricted nature of the PEFT subspace makes it inherently brittle. Because the model cannot "steer around" noise as effectively as FFT, it becomes highly susceptible to adversarial attacks. Our experiments on the AdvGLUE benchmark highlight this vulnerability.

Table 1: Performance comparison on GLUE and AdvGLUE benchmarks. While PEFT (e.g., LoRA) can match or exceed FFT on clean data (Origin), it suffers a significant performance drop on adversarial sets (Adv).

As shown in the table above, for the RoBbase model, FFT maintains an average adversarial accuracy of 36.58%, while LoRA drops to 30.89%—a gap of 5.69%. This "robustness collapse" indicates that PEFT solutions are less stable when faced with out-of-distribution perturbations.

Empirical Evidence: Weight Distribution

The theoretical constraints of PEFT are visible in the distribution of weight updates ($\Delta W$). Visual analysis shows that PEFT updates are highly concentrated around zero, whereas FFT updates are more widely distributed.

Figure 3: Distribution of weight updates (Delta Values) for different fine-tuning methods. FFT's broader distribution allows for more significant 're-wiring' of model logic compared to the concentrated updates of PEFT.

This wider distribution in FFT allows the model to undergo more significant structural changes, which is likely necessary for mastering complex tasks like the GSM8k math benchmark, where FFT-based LLaMA2-7B outperformed LoRA by 7.31%.

Conclusion

While PEFT is an invaluable tool for resource-constrained environments, it is not a perfect substitute for Full Fine-Tuning. The theoretical upper bounds and empirical evidence presented here suggest that PEFT is a "leaky abstraction" that eventually hits a wall in complex or adversarial settings. For tasks requiring high robustness or the learning of entirely new knowledge domains, Full Fine-Tuning remains the gold standard. Future research should look beyond established PEFT methods toward hybrid strategies that can bridge this capacity gap.

Summary updated

Abstract

Parameter-Efficient Fine-Tuning (PEFT) methods achieve performance comparable to Full Fine-Tuning (FFT) while requiring significantly fewer computing resources, making it the go-to choice for researchers. We find that although PEFT can achieve competitive results on some benchmarks, its performance falls short of FFT in complex tasks, such as reasoning and instruction-based fine-tuning. In this paper, we compare the characteristics of PEFT and FFT in terms of representational capacity and robustness based on optimization theory. We theoretically demonstrate that PEFT is a strict subset of FFT. By providing theoretical upper bounds for PEFT, we show that the limited parameter space constrains the model's representational ability, making it more susceptible to perturbations. Experiments on 15 datasets encompassing classification, generation, reasoning, instruction fine-tuning tasks and 11 adversarial test sets validate our theories. We hope that these results spark further research beyond the realms of well established PEFT. The source code is in the anonymous Github repository\footnote{https://github.com/misonsky/PEFTEval}.
BibTeX
@misc{liu2025look,
  author = {Yongkang Liu and Xingle Xu and Ercong Nie and Zijing Wang and Shi Feng and Daling Wang and Qian Li and Hinrich Schütze},
  title = {{Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning}},
  year = {2025},
  eprint = {2505.22355},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.22355},
}
Figure