WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling

Jiacheng Li, Jianchao Tan, Zhidong Yang, Pingwei Sun, Feiye Huo, Jiayu Qin, Yerui Sun, Yuchen Xie, Xunliang Cai, Xiangyu Zhang, Maoxin He, Guangming Tan, Weile Jia, Tong Zhao

Abstract

Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architectural modifications or optimizer adjustments. However, these approaches lack systematic optimization of weight patterns during training. Weight pattern refers to the distribution and relative magnitudes of weight parameters in a neural network. To address this issue, we propose a Weight Scaling method called WISCA to enhance training efficiency and model quality by strategically improving neural network weight patterns without changing network structures. By rescaling weights while preserving model outputs, WISCA indirectly optimizes the model's training trajectory. Experiments demonstrate that WISCA significantly improves convergence quality (measured by generalization capability and loss reduction), particularly in LLMs with Grouped Query Attention (GQA) architectures and LoRA fine-tuning tasks. Empirical results show 5.6% average improvement on zero-shot validation tasks and 2.12% average reduction in training perplexity across multiple architectures.

Introduction: The Challenge of Sharp Minima

Training Large Language Models (LLMs) is often a battle against the "sharp minimum" problem. In non-convex optimization, a model might converge to a point in the loss landscape where the "valley" is very narrow and steep. While the training loss might look good, these sharp minima generalize poorly to new data because slight shifts in the data distribution lead to massive jumps in error.

Most existing research focuses on changing the architecture (like adding layers) or tweaking the optimizer (like Adam). However, the authors argue we are ignoring weight patterns—the specific arrangement and relative magnitudes of parameters. Poorly structured weight patterns lead to rugged convergence paths. The challenge is: how do we move a model from a "sharp" region to a "flat" region of the loss landscape without destroying the progress it has already made during training?

Methods: Equivalent Model Theory

The authors introduce Equivalent Model Theory. Two models are "equivalent" if they have the same architecture and produce identical outputs for any given input, even if their internal weight values differ.

Conceptual diagram of WISCA jumping between equivalent models in the loss landscape to find flatter regions.

The core of their method, WISCA, identifies these equivalencies in Transformer components (like Attention) and LoRA layers. Specifically:

  1. Functional Equivalence: In self-attention, the output depends on the product of Query ($Q$) and Key ($K$) matrices. You can scale $W_q$ up and $W_k$ down by the same factor; the product $QK^T$ remains the same, but the "weight pattern" changes.
  2. Weight Rescaling: WISCA calculates the $L_1$-norm of weight matrices and rescales them so they are balanced (or follow a specific ratio for Grouped Query Attention).
  3. Two Flavors:
    • Tensor-wise: Global scaling applied to the entire matrix.
  4. Channel-wise: A finer-grained approach that scales at the individual channel level, which is particularly effective for handling the dimensional asymmetry in Grouped Query Attention (GQA).

By applying these transitions periodically during training or just at initialization, the model "jumps" to a flatter part of the loss landscape, leading to more stable and efficient convergence.

Key Results and Empirical Validation

The researchers evaluated WISCA across several models (TinyLlama, Qwen2-1.5B, Qwen1.5-MoE) and benchmarks. The results consistently show that WISCA configurations outperform the original baselines in both training loss and test perplexity.

Comparison of training loss and test perplexity across different models and WISCA strategies.
  • Generalization: On a 1.1B Llama model, the best WISCA configuration achieved a 10.1% average improvement over the baseline across zero-shot tasks.
  • Convergence Speed: In pre-training experiments, WISCA reduced training perplexity by an average of 2.12%.
  • LoRA Enhancement: When integrated with LoRA and PISSA, WISCA consistently lowered training loss, proving it is compatible with parameter-efficient fine-tuning.
  • Inference Acceleration: Applied to the EAGLE speculative decoding framework, WISCA increased the "acceptance rate" of draft tokens, directly translating to faster inference speeds.

Conclusion

WISCA demonstrates that the "Flatness" Principle—where regions with balanced weight magnitudes correspond to flatter loss geometry—can be leveraged to improve LLM training. Because the model remains functionally equivalent during the transition, WISCA can be injected into existing training pipelines with almost zero computational overhead. Future work may explore equivalent model strategies for other neural network structures beyond Transformers and more sophisticated metrics for judging the quality of a model's position in the loss landscape.

Summary updated

Abstract

Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architectural modifications or optimizer adjustments. However, these approaches lack systematic optimization of weight patterns during training. Weight pattern refers to the distribution and relative magnitudes of weight parameters in a neural network. To address this issue, we propose a Weight Scaling method called WISCA to enhance training efficiency and model quality by strategically improving neural network weight patterns without changing network structures. By rescaling weights while preserving model outputs, WISCA indirectly optimizes the model's training trajectory. Experiments demonstrate that WISCA significantly improves convergence quality (measured by generalization capability and loss reduction), particularly in LLMs with Grouped Query Attention (GQA) architectures and LoRA fine-tuning tasks. Empirical results show 5.6% average improvement on zero-shot validation tasks and 2.12% average reduction in training perplexity across multiple architectures.
BibTeX
@inproceedings{li2026wisca,
  author = {Jiacheng Li and Jianchao Tan and Zhidong Yang and Pingwei Sun and Feiye Huo and Jiayu Qin and Yerui Sun and Yuchen Xie and Xunliang Cai and Xiangyu Zhang and Maoxin He and Guangming Tan and Weile Jia and Tong Zhao},
  title = {{WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.79},
}
Figure