TL;DR
W2T (Weight-to-Token) is a novel framework that extracts downstream behavior and characteristics directly from LoRA adapter weights without running model inference or accessing training data. By using an efficient QR-SVD decomposition to eliminate factorization ambiguity, W2T enables standard Transformers to reliably perform tasks like attribute classification, performance prediction, and adapter retrieval.
What Problem Does This Paper Address?
With the explosion of Low-Rank Adaptation (LoRA) checkpoints in open-source repositories, managing, searching, and evaluating large collections of adapters has become a massive challenge. Traditional methods require running expensive large language model (LLM) inference or accessing the original training data, which is often missing or incomplete.
Why not just read the weights directly? The core obstacle is factorization non-uniqueness ($GL(r)$ reparameterization symmetry). A single weight update $\Delta W$ can be factorized into infinitely many pairs of low-rank matrices ($B, A$). Standard neural networks trained directly on these raw factors end up overfitting to the specific factorization style rather than learning the true underlying model behavior.
Methods
The authors introduce W2T (Weight-to-Token) to resolve this symmetry at the data level before feeding it into a model:
- Canonical Decomposition via QR-SVD: Instead of computing a costly Singular Value Decomposition (SVD) on the full reconstructed weight matrix $\Delta W$, W2T applies thin QR decomposition to the low-rank factors $B$ and $A^\top$. This reduces the problem to a tiny $r \times r$ core matrix, which is then decomposed via SVD and "lifted" back to form a provably unique, canonical representation.
- Modulated Tokenization: The resulting rank-wise singular vectors (input and output directions) are projected and fused into tokens. The singular values are injected via a stabilizing log-transform and MLP modulation, controlling how strongly each rank component expresses itself.
- Hierarchical Transformer Modeling: A rank-level Transformer processes tokens within each weight matrix position, followed by a $\sigma$-guided pooling mechanism. Finally, a position-level Transformer captures cross-position dependencies across the entire network to produce a global weight-space embedding.
Datasets
The framework was evaluated across both vision and language modalities using several LoRA checkpoint collections:
- CelebA-LoRA & CUB-LoRA: Vision adapters trained on Stable Diffusion v1.4 for concept/attribute modification.
- GoEmotions-LoRA: Language adapters trained on Llama-3.2-3B for multi-label emotion classification.
- ARC-Easy-LoRA: Language adapters trained on Llama-3.2-3B under randomized hyperparameter configurations to predict downstream accuracy.
- Mixed Language Adapter Pool: Checkpoints from ARC-Challenge, BoolQ, GSM8K, and MBPP used to evaluate zero-shot adapter retrieval.
Key Insights and Ideas
- Symmetry Resolution is the Secret Sauce: Resolving the continuous $GL(r)$ symmetry at the data level (via canonicalization) is far more effective and flexible than trying to enforce it via specialized equivariant network architectures.
- Weights Alone are Enough: LoRA weights inherently contain highly structured semantic and performance signals. Once the mathematical ambiguity is stripped away, standard architectures like Transformers can easily decode them.
- Computationally Elegant: The thin QR-SVD trick achieves a massive 93.18× speedup compared to direct SVD on the full matrix, making weight-space learning practical for large-scale model registries.
Key Results
- Attribute Classification: On the challenging GoEmotions-LoRA dataset, W2T achieved a Macro-F1 of 5.67% and AUROC of 61.25%, drastically outperforming raw-factor baselines (which scored near 0% Macro-F1) and the symmetry-aware baseline GLNet.
- Performance Prediction: W2T predicted downstream accuracy on ARC-LoRA with exceptional precision, achieving a Pearson correlation of 98.98% and a Mean Absolute Error (MAE) of just 0.32%.
- Adapter Retrieval: When transferring an encoder trained on ARC-Easy to retrieve adapters for entirely unseen tasks (like BoolQ or MBPP), W2T achieved an average NDCG@10 of 65.71%, vastly superior to standard raw cosine similarity or other learned baselines.
- Cross-Model Transfer: W2T demonstrated robust zero-shot transferability across different base models (e.g., training on SD1.4 adapters and evaluating on SD1.5), maintaining performance close to the in-domain oracle.
Limitations and Future Work
- Standalone Scope: W2T currently only evaluates standalone LoRA checkpoints attached to a fixed backbone. It does not yet handle merged adapters, composed checkpoints, or dynamic multi-adapter routing workflows.
- Discriminative Focus: The current work focuses entirely on discriminative tasks (classification, regression, retrieval). How this canonical representation transfers to generative weight-space tasks (e.g., generating new LoRA weights) remains an open question.
- Retrieval System Scaling: The retrieval experiments serve as a proxy setup; deploying this in a production-grade model registry would require further engineering around indexing, negative sampling, and serving infrastructure.
Summary updated