No article text was recorded for this version.
SVD-based weight initialization
Version 2 of 2204 papers297 conceptsfrozen 2026-10-04
Papers in this version
- Blog: HRP: High-Rank Preheating for Superior LoRA Initialization
- Blog: One-step full gradient suffices for low-rank fine-tuning, provably and efficiently
- Blog: PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
- Blog: Geometry-Preserving Orthonormal Initialization for Low-Rank Adaptation in RLVR
- Blog: MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning
- Blog: One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
- Blog: The Primacy of Magnitude in Low-Rank Adaptation
- Blog: GoRA: Gradient-driven Adaptive Low Rank Adaptation
- Blog: Mitigating Error Propagation in Low-Rank Approximation of Large Models via Distribution-Aware Whitening
- Blog: PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
- Blog: Dual Decomposition of Weights and Singular Value Low Rank Adaptation
- Blog: ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints
- Blog: Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning
- Blog: NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models
- Blog: Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning
- Blog: 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
- Blog: LoRA-GA: Low-Rank Adaptation with Gradient Approximation
- Blog: LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently
- Blog: LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis
- Blog: LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters
- Blog: Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models
- Blog: TuneComp: Joint Fine-tuning and Compression for Large Foundation Models
- Blog: OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation
- Blog: Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
- Blog: KeepLoRA: Continual Learning with Residual Gradient Adaptation
- Blog: ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints
- Blog: Unleashing the Power of Task-Specific Directions in Parameter Efficient Fine-tuning
- Blog: Pufferfish: Communication-efficient Models At No Extra Cost
- Blog: SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation
- Blog: A new low-rank adaptation method for brain structure and metastasis segmentation via decoupled principal weight direction and magnitude
- Blog: LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation
- Blog: Domain Adaptation Using Factorized Hidden Layer for Robust Automatic Speech Recognition
- Blog: Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation
- Blog: Efficient Low-Dimensional Compression of Overparameterized Models
- Blog: Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization
- Blog: LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model
- Blog: LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter Sharing
- Blog: LOST: Low-rank and Sparse Pre-training for Large Language Models
- Blog: LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation
- Blog: SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation
- Blog: LoQT: Low Rank Adapters for Quantized Training
- Blog: Encoding-based Memory Modules for Recurrent Neural Networks
- Blog: IPA: An Information-Reconstructive Input Projection Framework for Efficient Foundation Model Adaptation
- Blog: Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning
- Blog: GoRA: Gradient-driven Adaptive Low Rank Adaptation
- Blog: HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DST
- Blog: Few-Shot Data-Driven Algorithms for Low Rank Approximation
- Blog: CoLA: Collaborative Low-Rank Adaptation
- Blog: MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning
- Blog: Efficient Algorithms for the Hadamard Decomposition
- Blog: CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
- Blog: Efficient Multi-Domain Network Learning by Covariance Normalization
- Blog: CoLA: Collaborative Low-Rank Adaptation
- Blog: SumRA: Parameter Efficient Fine-tuning with Singular Value Decomposition and Summed Orthogonal Basis
- Blog: Parameter Efficient Fine-tuning via Explained Variance Adaptation
- Blog: HRP: High-Rank Preheating for Superior LoRA Initialization
- Blog: X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression
- Blog: GradMax: Growing Neural Networks using Gradient Information
- Blog: LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
- Blog: ABBA-Adapters: Efficient and Expressive Fine-Tuning of Foundation Models
Showing 60 of 204 papers.
History & citation
Current · Permanent link
- Gradient-Informed Initialization for Low-Rank Adaptation merged into this direction (100% overlap)
- Informed Initialization for Low-Rank Adaptation merged into this direction (73% overlap)
Cite this direction
Latest version (a living page):
@misc{lacuna_svd_based_weight_initialization_3fdce1c1,
title = {SVD-based weight initialization},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/svd-based-weight-initialization/rdr_3fdce1c144964917b0abba7e14d0d6b9},
note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}This exact version (version 2, frozen text):
@misc{lacuna_svd_based_weight_initialization_3fdce1c1_v2,
title = {SVD-based weight initialization},
author = {{Lacuna}},
howpublished = {Lacuna research direction},
year = {2026},
url = {https://lacuna.tiptreesystems.com/direction/svd-based-weight-initialization/rdr_3fdce1c144964917b0abba7e14d0d6b9/v/2},
note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}