Direction or Magnitude? Learning Scalars over Task-Calibrated Directions

Ziqing Wen, Xiaohui Song, Jiahuan Wang, Ping Luo, Junlin Zeng, Dongsheng Li, Kele Xu, Tao SunarXiv 2026 · arXiv

TL;DR

SCALAR is a new parameter-efficient fine-tuning (PEFT) method that achieves LoRA-level performance while using up to 96% fewer trainable parameters. By learning only layer-specific scalar coefficients over fixed, task-calibrated directions, it reduces peak training memory by 20% and increases throughput, making high-quality model adaptation feasible on resource-constrained hardware.

What Problem Does This Paper Address?

While Large Language Models (LLMs) are powerful, fine-tuning them is notoriously expensive. Standard Low-Rank Adaptation (LoRA) significantly reduces costs, but as models grow, even LoRA’s memory footprint—driven by gradients and optimizer states for its trainable matrices—can be a bottleneck for on-device personalization or massive-scale multitasking.

The authors ask a fundamental question: Is it the direction or the magnitude of the weight update that matters most? They observed that if you take a trained rank-1 LoRA adapter, freeze its update direction, and re-optimize just a single scalar "magnitude" coefficient, you recover almost all the performance. This suggests that if we can pre-calibrate the right "directions" for a task, we can drastically simplify the learning process to just tuning scalars.

Methods

The proposed method, SCALAR (Subspace-Calibrated Adaptation via Layerwise Atom Reweighting), operates in three main steps:

  1. Task-Aware Input Directions: Before training, the model processes a small "calibration" set of task-related data. SCALAR performs a Singular Value Decomposition (SVD) on the resulting activations to find the dominant directions (the "input basis"). These directions are frozen.
  2. Grouped Random Output Dictionaries: For each input direction, SCALAR assigns a group of $H$ fixed, random output "atoms." These atoms form a dictionary that defines the potential output space for that specific input direction.
  3. Scalar Learning: During fine-tuning, the only trainable parameters are the scalar coefficients ($\gamma$) used to weight these random atoms.

To keep computation efficient, SCALAR uses grouped computation: it aggregates the weighted output atoms into a single effective vector before applying them to token activations. This ensures the actual matrix multiplication remains low-rank (rank $R$), regardless of how many atoms ($H$) are in the dictionary. Furthermore, compatible layers can share the same frozen random dictionaries to save storage.

Datasets

The authors evaluated SCALAR across a wide array of tasks and model families:

  • Commonsense Reasoning: 8 benchmarks (including BoolQ, PIQA, and HellaSwag) using the Commonsense170K dataset.
  • Mathematical Reasoning: Evaluated on GSM8K and MATH after training on MetaMath.
  • Code Generation: Evaluated on HumanEval and MBPP after training on CodeFeedback.
  • General Knowledge: Evaluated on MMLU following instruction tuning on the Alpaca-GPT4 dataset.
  • Models: Tested on Qwen2.5 (up to 32B), LLaMA3/3.2, and Gemma3.

Key Insights and Ideas

  • The Power of Magnitudes: The "aha moment" came from a diagnostic showing that tuning just 196 scalar parameters on a 7B model (compared to LoRA's 2.5 million) achieved nearly identical accuracy. This proves that the "direction" of useful perturbations is often inherent in the task activations themselves.
  • Activation-SVD is Theoretically Optimal: The paper provides theoretical proofs (Theorems 3.5 and 3.6) showing that an input basis derived from activation SVD captures the most "energy" and provides a stronger initial gradient signal than random bases.
  • Decoupling Parameters from Model Width: In LoRA, the number of parameters scales with the layer dimensions ($m+n$). In SCALAR, the number of trainable parameters ($RH$) is independent of the model's hidden dimension, allowing it to become proportionally more efficient as models get wider.

Key Results

SCALAR consistently matches or exceeds LoRA's performance while being significantly leaner:

  • Parameter Efficiency: On commonsense reasoning, SCALAR achieves accuracy comparable to rank-8 LoRA while using only ~4% of its trainable parameters.
  • System Gains: Measurements on a Qwen2.5-7B model show that SCALAR uses 20% lower peak training memory and provides 6% higher throughput than LoRA.
Figure 1: Average accuracy on 8 commonsense reasoning tasks across trainable parameter budgets. SCALAR approaches LoRA-level accuracy with substantially fewer trainable parameters.
  • Memory & Speed: As shown in the bar charts below, SCALAR (far right) maintains the lowest GPU memory usage and the highest token throughput among all evaluated PEFT methods.
Figure 2: Memory and throughput for fine-tuning Qwen2.5-7B on commonsense reasoning. We report peak GPU memory with a device batch size of 16 and token throughput on 2 NVIDIA H20 GPUs. SCALAR uses less memory and achieves higher throughput than the compared baselines.

Limitations and Future Work

  • Calibration Step: SCALAR requires a one-time pre-training calibration pass to collect activation statistics. While the authors introduced a "Fast SVD" to make this quick (reducing eigensolve time by ~19x), it remains an extra step compared to "plug-and-play" LoRA.
  • Fixed Update Space: Because the input basis is frozen after calibration, the model cannot "discover" entirely new input directions during training, though the random output dictionaries provide significant flexibility.
  • Future Directions: The authors suggest exploring adaptive rank allocation (changing $R$ per layer) and extending the method to non-linear components of the transformer block.

Summary updated

Abstract

Parameter-efficient fine-tuning (PEFT) aims to preserve adaptation quality while reducing training costs. We propose SCALAR, which achieves performance comparable to LoRA with substantially fewer trainable parameters, lower peak training memory, and higher throughput. SCALAR constructs a fixed input basis from task activations and assigns a random output dictionary to each input direction. Fine-tuning learns only layer-specific scalar coefficients, reducing gradient and optimizer-state storage. Grouped computation aggregates output atoms before applying the adapter to token activations, while dictionary sharing across compatible layers reduces fixed-basis storage. Experiments on commonsense reasoning, mathematical reasoning, code generation, and knowledge evaluation after instruction tuning demonstrate effective adaptation at small trainable budgets. On commonsense reasoning, SCALAR achieves accuracy comparable to LoRA with about 4% of its trainable parameters. System measurements on Qwen2.5-7B show 20% lower peak training memory and 6% higher throughput than LoRA. The combination of adaptation quality, a compact trainable state, and training efficiency makes SCALAR a promising approach to resource-constrained fine-tuning.
BibTeX
@misc{wen2026direction,
  author = {Ziqing Wen and Xiaohui Song and Jiahuan Wang and Ping Luo and Junlin Zeng and Dongsheng Li and Kele Xu and Tao Sun},
  title = {{Direction or Magnitude? Learning Scalars over Task-Calibrated Directions}},
  year = {2026},
  eprint = {2611.17784},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17784},
}
Figure