Abstract
Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architectural modifications or optimizer adjustments. However, these approaches lack systematic optimization of weight patterns during training. Weight pattern refers to the distribution and relative magnitudes of weight parameters in a neural network. To address this issue, we propose a Weight Scaling method called WISCA to enhance training efficiency and model quality by strategically improving neural network weight patterns without changing network structures. By rescaling weights while preserving model outputs, WISCA indirectly optimizes the model's training trajectory. Experiments demonstrate that WISCA significantly improves convergence quality (measured by generalization capability and loss reduction), particularly in LLMs with Grouped Query Attention (GQA) architectures and LoRA fine-tuning tasks. Empirical results show 5.6% average improvement on zero-shot validation tasks and 2.12% average reduction in training perplexity across multiple architectures.
Introduction: The Challenge of Sharp Minima
Training Large Language Models (LLMs) is often a battle against the "sharp minimum" problem. In non-convex optimization, a model might converge to a point in the loss landscape where the "valley" is very narrow and steep. While the training loss might look good, these sharp minima generalize poorly to new data because slight shifts in the data distribution lead to massive jumps in error.
Most existing research focuses on changing the architecture (like adding layers) or tweaking the optimizer (like Adam). However, the authors argue we are ignoring weight patterns—the specific arrangement and relative magnitudes of parameters. Poorly structured weight patterns lead to rugged convergence paths. The challenge is: how do we move a model from a "sharp" region to a "flat" region of the loss landscape without destroying the progress it has already made during training?
Methods: Equivalent Model Theory
The authors introduce Equivalent Model Theory. Two models are "equivalent" if they have the same architecture and produce identical outputs for any given input, even if their internal weight values differ.
The core of their method, WISCA, identifies these equivalencies in Transformer components (like Attention) and LoRA layers. Specifically:
- Functional Equivalence: In self-attention, the output depends on the product of Query ($Q$) and Key ($K$) matrices. You can scale $W_q$ up and $W_k$ down by the same factor; the product $QK^T$ remains the same, but the "weight pattern" changes.
- Weight Rescaling: WISCA calculates the $L_1$-norm of weight matrices and rescales them so they are balanced (or follow a specific ratio for Grouped Query Attention).
- Two Flavors:
- Tensor-wise: Global scaling applied to the entire matrix.
- Channel-wise: A finer-grained approach that scales at the individual channel level, which is particularly effective for handling the dimensional asymmetry in Grouped Query Attention (GQA).
By applying these transitions periodically during training or just at initialization, the model "jumps" to a flatter part of the loss landscape, leading to more stable and efficient convergence.
Key Results and Empirical Validation
The researchers evaluated WISCA across several models (TinyLlama, Qwen2-1.5B, Qwen1.5-MoE) and benchmarks. The results consistently show that WISCA configurations outperform the original baselines in both training loss and test perplexity.
- Generalization: On a 1.1B Llama model, the best WISCA configuration achieved a 10.1% average improvement over the baseline across zero-shot tasks.
- Convergence Speed: In pre-training experiments, WISCA reduced training perplexity by an average of 2.12%.
- LoRA Enhancement: When integrated with LoRA and PISSA, WISCA consistently lowered training loss, proving it is compatible with parameter-efficient fine-tuning.
- Inference Acceleration: Applied to the EAGLE speculative decoding framework, WISCA increased the "acceptance rate" of draft tokens, directly translating to faster inference speeds.
Conclusion
WISCA demonstrates that the "Flatness" Principle—where regions with balanced weight magnitudes correspond to flatter loss geometry—can be leveraged to improve LLM training. Because the model remains functionally equivalent during the transition, WISCA can be injected into existing training pipelines with almost zero computational overhead. Future work may explore equivalent model strategies for other neural network structures beyond Transformers and more sophisticated metrics for judging the quality of a model's position in the loss landscape.
Summary updated