Graph-Based Spectral Decomposition for Parameter Coordination in Language Model Fine-Tuning

Hanlu Zhang, Yumeng Ma, Shuo Wang, Guiran Liu, Binrong Zhu

TL;DR

This paper introduces a novel fine-tuning framework for Large Language Models (LLMs) that treats model parameters as nodes in a graph. By applying spectral graph theory, the authors coordinate parameter updates in the frequency domain, resulting in a "structurally aware" optimization process that outperforms standard methods like LoRA and full-parameter fine-tuning in both accuracy and convergence speed.

What Problem Does This Paper Address?

Fine-tuning LLMs is notoriously difficult. Standard approaches usually fall into two camps: Full-Tuning, which is computationally ruinous for billion-parameter models, and Parameter-Efficient Fine-Tuning (PEFT) (like LoRA or Adapters), which reduces costs but often ignores the complex, latent structural relationships between parameters.

The authors argue that current optimization strategies treat parameters as isolated values or simple statistical distributions. This "blindness" to the underlying architecture leads to slow convergence, instability, and a loss of the model's original structural integrity during adaptation. The challenge is to find a way to update parameters collaboratively, ensuring that an update in one part of the model is "aware" of its relationship to others.

Methods

The authors propose a Graph-Based Spectral Decomposition approach. Here is the step-by-step breakdown of their "Spectral Collaborative Optimization":

  1. Graph Construction: They treat the parameters of a pre-trained model as nodes in a weighted graph. Edges represent the dependencies or coordination relationships between these parameters.
  2. Spectral Mapping: Using the Graph Laplacian ($L = D - W$), they perform spectral decomposition to move the parameters from the "spatial" domain into the frequency domain. This reveals the global structural relationships of the parameter space.
  3. Joint Loss Function: Instead of just optimizing for the task at hand, they introduce a spectral regularization term. The final loss is a balance:
  4. Task Loss: How well the model performs the new job.
  5. Spectral Regularization: How well the parameter updates preserve the model's essential structural "vibe."
  6. Spectral Filtering: During the gradient update phase, they apply a low-pass filter. This suppresses high-frequency noise (erratic, unstable updates) while retaining the low-frequency signals that represent the core structural information of the graph.

Datasets

The researchers evaluated their method using the OpenWebText dataset, a high-quality English corpus scraped from Reddit-linked web pages.

  • Scale: They extracted a 20GB subset.
  • Diversity: The data covers technology, medicine, economics, and education.
  • Preprocessing: They used advanced cleaning techniques (contrastive and variational frameworks) to remove HTML artifacts and ensure coherent paragraph flow, creating a 9:1 training/validation split.

Key Insights and Ideas

  • Parameters as a Signal: The most innovative "aha" moment is treating the weights of a neural network as a signal living on a graph. This allows researchers to use tools from Digital Signal Processing (DSP)—like filters—to "clean up" the learning process.
  • Global Coordination over Local Updates: Traditional SGD updates weights somewhat independently. By using spectral information, this method enables cross-layer coordination, allowing the model to avoid local optima that plague layer-wise updates.
  • Structure as a Prior: By regularizing in the spectral domain, the model treats its pre-trained structure as a "prior" that shouldn't be recklessly discarded. This leads to much better performance in few-shot learning scenarios where data is scarce.

Key Results

The proposed method was tested against Full-Tuning, Adapters, and LoRA. The results were consistently superior:

  • Accuracy: The spectral method achieved 91.47%, significantly higher than LoRA (89.02%) and Full-Tuning (88.23%).
  • Recall: It showed a major jump in recall (90.32%), suggesting the model is much better at capturing the full breadth of the target task.
  • Few-Shot Generalization: In tests with only 20% of the training data, the spectral method maintained nearly 80% accuracy, while the Adapter method struggled significantly.
  • Convergence: The joint loss function reached a lower final loss value faster than using task loss alone, proving that structural constraints actually accelerate learning rather than hindering it.

Limitations and Future Work

While the results are promising, the paper hints at a few areas for further exploration:

  • Static vs. Dynamic Graphs: The current graph structure is constructed based on pre-trained dependencies. The authors suggest that dynamic graph generation—where the graph evolves as the model learns—could be a powerful next step.
  • Computational Overhead: While the fine-tuning itself is efficient, the initial spectral decomposition of a massive parameter graph can be computationally expensive. Future work may need to focus on approximation methods for even larger "ultra-scale" models.
  • Multimodal Expansion: The authors intend to extend this framework to vision-language models and other complex architectures to see if the spectral benefits translate across modalities.

Summary updated

Abstract

This paper proposes a parameter collaborative optimization algorithm for large language models, enhanced with graph spectral analysis. The goal is to improve both fine-tuning efficiency and structural awareness during training. In the proposed method, the parameters of a pre-trained language model are treated as nodes in a graph. A weighted graph is constructed, and Laplacian spectral decomposition is applied to enable frequency-domain modeling and structural representation of the parameter space. Based on this structure, a joint loss function is designed. It combines the task loss with a spectral regularization term to facilitate collaborative updates among parameters. In addition, a spectral filtering mechanism is introduced during the optimization phase. This mechanism adjusts gradients in a structure-aware manner, enhancing the model's training stability and convergence behavior. The method is evaluated on multiple tasks, including traditional fine-tuning comparisons, few-shot generalization tests, and convergence speed analysis. In all settings, the proposed approach demonstrates superior performance. The experimental results confirm that the spectral collaborative optimization framework effectively reduces parameter perturbations and improves fine-tuning quality while preserving overall model performance. This work contributes significantly to the field of artificial intelligence by advancing parameter-efficient training methodologies for large-scale models, reinforcing the importance of structural signal processing in deep learning optimization, and offering a robust, generalizable framework for enhancing language model adaptability and performance.
BibTeX
@misc{zhang2025graph,
  author = {Hanlu Zhang and Yumeng Ma and Shuo Wang and Guiran Liu and Binrong Zhu},
  title = {{Graph-Based Spectral Decomposition for Parameter Coordination in Language Model Fine-Tuning}},
  year = {2025},
  eprint = {2504.19583},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.19583},
  howpublished = {arXiv},
}
Figure