WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling

Jiacheng Li, Jianchao Tan, Zhidong Yang, Pingwei Sun, Feiye Huo, Jiayu Qin, Xiangyu Zhang, Maoxin He, Guangming Tan, Weile Jia, Xunliang Cai, Tong Zhao

TL;DR

WISCA (Weight Scaling) is a lightweight optimization strategy that improves LLM training by transitioning models to "equivalent" versions with better weight distributions. By strategically rescaling weights to favor flatter regions of the loss landscape without altering model output, WISCA achieves a 5.6% average improvement in zero-shot tasks and a 2.12% reduction in training perplexity across various architectures.

What Problem Does This Paper Address?

Training Large Language Models (LLMs) is often a battle against the "sharp minimum" problem. In non-convex optimization, a model might converge to a point in the loss landscape where the "valley" is very narrow and steep. While the training loss might look good, these sharp minima generalize poorly to new data because slight shifts in the data distribution lead to massive jumps in error.

Most existing research focuses on changing the architecture (like adding layers) or tweaking the optimizer (like Adam). However, the authors argue we are ignoring weight patterns—the specific arrangement and relative magnitudes of parameters. Poorly structured weight patterns lead to rugged convergence paths. The challenge is: how do we move a model from a "sharp" region to a "flat" region of the loss landscape without destroying the progress it has already made during training?

Methods

The authors introduce Equivalent Model Theory. Two models are "equivalent" if they have the same architecture and produce identical outputs for any given input, even if their internal weight values differ.

The core of their method, WISCA, identifies these equivalencies in Transformer components (like Attention) and LoRA layers. Specifically:

  1. Functional Equivalence: In self-attention, the output depends on the product of Query ($Q$) and Key ($K$) matrices. You can scale $W_q$ up and $W_k$ down by the same factor; the product $QK^T$ remains the same, but the "weight pattern" changes.
  2. Weight Rescaling: WISCA calculates the $L_1$-norm of weight matrices and rescales them so they are balanced (or follow a specific ratio for Grouped Query Attention).
  3. Two Flavors:
    • Tensor-wise: Global scaling applied to the entire matrix.
  4. Channel-wise: A finer-grained approach that scales at the individual channel level, which is particularly effective for handling the dimensional asymmetry in Grouped Query Attention (GQA).

By applying these transitions periodically during training (e.g., every 250 iterations) or just at initialization, the model "jumps" to a flatter part of the loss landscape, leading to more stable and efficient convergence.

Datasets

The researchers evaluated WISCA across several models and benchmarks:

  • Pre-training: TinyLlama, Qwen2-1.5B, and Qwen1.5-MoE trained on the TinyStories and Wikipedia (en) datasets.
  • Fine-tuning (PEFT): DeepSeek-V2-Lite-Chat fine-tuned on Alpaca (instruction following) and MetaMath (mathematical reasoning).
  • Speculative Decoding: EAGLE draft models trained on the ShareGPT dataset.
  • Evaluation: Zero-shot tasks via the EleutherAI LM Evaluation Harness (including BoolQ, PIQA, ARC, and HellaSwag).

Key Insights and Ideas

  • The "Flatness" Principle: The authors demonstrate that the region where $Q = K$ (in terms of magnitude/scaling) corresponds to the flattest geometry in the loss landscape. WISCA essentially "pushes" the model toward this symmetry.
  • GQA is Naturally Imbalanced: In Grouped Query Attention, there are more Query heads than Key/Value groups. This inherent imbalance creates a significant "weight pattern" problem that WISCA is uniquely suited to fix via group-averaged normalization.
  • Synergy in Scaling: The best results come from scaling both the $Q/K$ pair (attention scores) and the $V/O$ pair (output projection) simultaneously. The improvements are more than the sum of their parts.
  • "Free" Gains: Because the model remains functionally equivalent during the transition, WISCA can be injected into existing training pipelines with almost zero computational overhead.

Key Results

  • Generalization: On a 1.1B Llama model, the best WISCA configuration achieved a 10.1% average improvement over the baseline across zero-shot tasks.
  • Convergence Speed: In pre-training experiments, WISCA reduced training perplexity by an average of 2.12%.
  • LoRA Enhancement: When integrated with LoRA and PISSA, WISCA consistently lowered training loss, proving it is compatible with parameter-efficient fine-tuning.
  • Inference Acceleration: Applied to the EAGLE speculative decoding framework, WISCA increased the "acceptance rate" of draft tokens, directly translating to faster inference speeds.

Limitations and Future Work

  • Frequency of Adjustment: While "initialization-only" WISCA captures about 97% of the gains, the authors found that periodic adjustments provide incremental refinements. However, they don't provide a definitive "optimal frequency" for all architectures.
  • Scope of Equivalence: The current work focuses on linear layers and attention mechanisms. The authors suggest that future work could explore equivalent model strategies for other neural network structures beyond Transformers.
  • Judging Convergence: The paper notes that while they use weight patterns to improve convergence, more sophisticated metrics for "judging" the quality of a model's current position in the loss landscape are still needed.

Summary updated

Abstract

Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architectural modifications or optimizer adjustments. However, these approaches lack systematic optimization of weight patterns during training. Weight pattern refers to the distribution and relative magnitudes of weight parameters in a neural network. To address this issue, we propose a Weight Scaling method called WISCA to enhance training efficiency and model quality by strategically improving neural network weight patterns without changing network structures. By rescaling weights while preserving model outputs, WISCA indirectly optimizes the model's training trajectory. Experiments demonstrate that WISCA significantly improves convergence quality (measured by generalization capability and loss reduction), particularly in LLMs with Grouped Query Attention (GQA) architectures and LoRA fine-tuning tasks. Empirical results show 5.6% average improvement on zero-shot validation tasks and 2.12% average reduction in training perplexity across multiple architectures.
BibTeX
@inproceedings{li2026wisca,
  author = {Jiacheng Li and Jianchao Tan and Zhidong Yang and Pingwei Sun and Feiye Huo and Jiayu Qin and Xiangyu Zhang and Maoxin He and Guangming Tan and Weile Jia and Xunliang Cai and Tong Zhao},
  title = {{WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.79},
}

Loading related content...

Figure