W2T: LoRA Weights Already Know What They Can Do

Xiaolong Han, Ferrante Neri, Zijian Jiang, Fang Wu, Yanfang Ye, Lu Yin, Zehong WangarXiv 2026 · arXiv

TL;DR

W2T (Weight-to-Token) is a novel framework that extracts downstream behavior and characteristics directly from LoRA adapter weights without running model inference or accessing training data. By using an efficient QR-SVD decomposition to eliminate factorization ambiguity, W2T enables standard Transformers to reliably perform tasks like attribute classification, performance prediction, and adapter retrieval.

What Problem Does This Paper Address?

With the explosion of Low-Rank Adaptation (LoRA) checkpoints in open-source repositories, managing, searching, and evaluating large collections of adapters has become a massive challenge. Traditional methods require running expensive large language model (LLM) inference or accessing the original training data, which is often missing or incomplete.

Why not just read the weights directly? The core obstacle is factorization non-uniqueness ($GL(r)$ reparameterization symmetry). A single weight update $\Delta W$ can be factorized into infinitely many pairs of low-rank matrices ($B, A$). Standard neural networks trained directly on these raw factors end up overfitting to the specific factorization style rather than learning the true underlying model behavior.

Figure 1: LoRA updates already encode downstream behaviors, but reading them requires a symmetry- aware representation. (a) LoRA checkpoints trained from raw data are treated as data objects for downstream tasks. (b) Symmetry-agnostic processing of raw LoRA factors. (c) Symmetry-aware decomposition of LoRA updates into canonical components.

Methods

The authors introduce W2T (Weight-to-Token) to resolve this symmetry at the data level before feeding it into a model:

  1. Canonical Decomposition via QR-SVD: Instead of computing a costly Singular Value Decomposition (SVD) on the full reconstructed weight matrix $\Delta W$, W2T applies thin QR decomposition to the low-rank factors $B$ and $A^\top$. This reduces the problem to a tiny $r \times r$ core matrix, which is then decomposed via SVD and "lifted" back to form a provably unique, canonical representation.
  2. Modulated Tokenization: The resulting rank-wise singular vectors (input and output directions) are projected and fused into tokens. The singular values are injected via a stabilizing log-transform and MLP modulation, controlling how strongly each rank component expresses itself.
  3. Hierarchical Transformer Modeling: A rank-level Transformer processes tokens within each weight matrix position, followed by a $\sigma$-guided pooling mechanism. Finally, a position-level Transformer captures cross-position dependencies across the entire network to produce a global weight-space embedding.
Overview of the W2T framework

Datasets

The framework was evaluated across both vision and language modalities using several LoRA checkpoint collections:

  • CelebA-LoRA & CUB-LoRA: Vision adapters trained on Stable Diffusion v1.4 for concept/attribute modification.
  • GoEmotions-LoRA: Language adapters trained on Llama-3.2-3B for multi-label emotion classification.
  • ARC-Easy-LoRA: Language adapters trained on Llama-3.2-3B under randomized hyperparameter configurations to predict downstream accuracy.
  • Mixed Language Adapter Pool: Checkpoints from ARC-Challenge, BoolQ, GSM8K, and MBPP used to evaluate zero-shot adapter retrieval.

Key Insights and Ideas

  • Symmetry Resolution is the Secret Sauce: Resolving the continuous $GL(r)$ symmetry at the data level (via canonicalization) is far more effective and flexible than trying to enforce it via specialized equivariant network architectures.
  • Weights Alone are Enough: LoRA weights inherently contain highly structured semantic and performance signals. Once the mathematical ambiguity is stripped away, standard architectures like Transformers can easily decode them.
  • Computationally Elegant: The thin QR-SVD trick achieves a massive 93.18× speedup compared to direct SVD on the full matrix, making weight-space learning practical for large-scale model registries.

Key Results

  • Attribute Classification: On the challenging GoEmotions-LoRA dataset, W2T achieved a Macro-F1 of 5.67% and AUROC of 61.25%, drastically outperforming raw-factor baselines (which scored near 0% Macro-F1) and the symmetry-aware baseline GLNet.
  • Performance Prediction: W2T predicted downstream accuracy on ARC-LoRA with exceptional precision, achieving a Pearson correlation of 98.98% and a Mean Absolute Error (MAE) of just 0.32%.
  • Adapter Retrieval: When transferring an encoder trained on ARC-Easy to retrieve adapters for entirely unseen tasks (like BoolQ or MBPP), W2T achieved an average NDCG@10 of 65.71%, vastly superior to standard raw cosine similarity or other learned baselines.
Figure 3: Local retrieval neighborhood for a 8-shot BoolQ query. Each subplot shows the same query (star) amid a shared pool. W2T places relevant adapters (dark) closest to the query with top-1 hit and rel@5=5, while baselines surface more irrelevant neighbors.
  • Cross-Model Transfer: W2T demonstrated robust zero-shot transferability across different base models (e.g., training on SD1.4 adapters and evaluating on SD1.5), maintaining performance close to the in-domain oracle.

Limitations and Future Work

  • Standalone Scope: W2T currently only evaluates standalone LoRA checkpoints attached to a fixed backbone. It does not yet handle merged adapters, composed checkpoints, or dynamic multi-adapter routing workflows.
  • Discriminative Focus: The current work focuses entirely on discriminative tasks (classification, regression, retrieval). How this canonical representation transfers to generative weight-space tasks (e.g., generating new LoRA weights) remains an open question.
  • Retrieval System Scaling: The retrieval experiments serve as a proxy setup; deploying this in a production-grade model registry would require further engineering around indexing, negative sampling, and serving infrastructure.

Summary updated

Abstract

Each LoRA checkpoint compactly stores task-specific updates in low-rank weight matrices, offering an efficient way to adapt large language models to new tasks and domains. In principle, these weights already encode what the adapter does and how well it performs. In this paper, we ask whether this information can be read directly from the weights, without running the base model or accessing training data. A key obstacle is that a single LoRA update can be factorized in infinitely many ways. Without resolving this ambiguity, models trained on the factors may fit the particular factorization rather than the underlying update. To this end, we propose \methodfull, which maps each LoRA update to a provably canonical form via QR decomposition followed by SVD, so that all equivalent factorizations share the same representation. The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding. Across language and vision LoRA collections, W2T achieves strong results on attribute classification, performance prediction, and adapter retrieval, demonstrating that LoRA weights reliably indicate model behavior once factorization ambiguity is removed. Code is available at https://github.com/xiaolonghan2000/Weight2Token.
BibTeX
@misc{han2026w,
  author = {Xiaolong Han and Ferrante Neri and Zijian Jiang and Fang Wu and Yanfang Ye and Lu Yin and Zehong Wang},
  title = {{W2T: LoRA Weights Already Know What They Can Do}},
  year = {2026},
  eprint = {2603.15990},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.15990},
}
Figure