Abstract
Training deep neural networks (DNNs) in low-dimensional subspaces is a promising direction for achieving efficient training and better generalization performance. Previous works extract the subspaces by using random projection or performing dimensionality reduction method on the training trajectory, but these methods can be inefficient or unstable in terms of dimensionality and numerical operations. In this paper, we connect subspace training to weight averaging and propose Trainable Weight Averaging (TWA), a general approach for subspace training that generalizes the previous efforts. TWA is efficient in terms of dimensionality and also easy to use, making it a promising new method for subspace training. We further design an efficient scheme for subspace training to cope with large-scale problems, which allows parallel training across multiple nodes and evenly distributing the memory and computation burden to each node. We apply TWA to efficient neural network training and improving fine-tuning performance tasks to demonstrate the great efficiency and effectiveness of our approach. We conduct extensive experiments that cover various benchmark computer vision and neural language processing tasks with various architectures.
Introduction
Weight averaging has become a standard technique in deep learning to improve model generalization by smoothing the loss landscape. Methods like Stochastic Weight Averaging (SWA) and Exponential Moving Averages (EMA) are widely used but rely on fixed or heuristic weighting schemes. For instance, SWA typically assigns equal weight to all checkpoints in a trajectory. However, these fixed schemes are often suboptimal because checkpoints at different stages of training contribute differently to the final performance.
To address these limitations, we introduce Trainable Weight Averaging (TWA). TWA transforms weight averaging into an optimization problem within a low-dimensional subspace. By learning the optimal coefficients for combining model checkpoints, TWA achieves faster convergence and superior generalization compared to traditional fixed-weight methods.
Methodology
The core of TWA is the conceptualization of weight averaging as a subspace training task. Instead of optimizing in the high-dimensional full-parameter space $\mathbb{R}^D$, TWA operates in a tiny subspace $\mathbb{R}^n$ spanned by $n$ candidate weight checkpoints.
Subspace Construction and Optimization
TWA constructs a set of basis vectors from historical model checkpoints. To ensure stability and speed, these checkpoints are decoupled using decentralization and normalization, a process significantly more efficient than traditional Gram-Schmidt orthogonalization.
We propose two primary variants:
- TWA-t: Optimizes coefficients using the training dataset.
- TWA-v: Uses a small, held-out validation set (e.g., 2% of training data). TWA-v acts as a powerful regularizer, finding the combination of weights that generalizes best to unseen data.
As shown in the comparison above, TWA differs from methods like SWA or EMA by defining its solution set as the span of checkpoints rather than a fixed average. This allows for a higher-dimensional solution space ($n$ or $nL$ for layer-wise TWA) while maintaining $O(1)$ inference cost, unlike traditional ensembles which scale linearly with the number of models.
Efficiency and Scalability
To handle large-scale models, TWA incorporates several efficiency-focused designs:
- Distributed Training: The projection matrix is partitioned across multiple GPUs to prevent memory bottlenecks.
- Low-Bit Quantization: The projection matrix can be quantized to 4 bits with negligible performance loss, reducing memory usage by 8x.
- Layer-wise Processing: TWA can learn independent coefficients for different network layers, providing finer control over the model's composition.
Experimental Results
We evaluated TWA across diverse tasks, including image classification (CIFAR, ImageNet), machine translation (WMT2014), and language modeling (WikiText-2), using architectures ranging from ResNets to Transformers.
Training Acceleration and Generalization
TWA significantly reduces training time. On CIFAR-100, we observed a 45% reduction in total training time. On ImageNet, TWA achieved a 32% reduction while maintaining or exceeding baseline accuracy. Notably, for ViT-B/16 on ImageNet, TWA-v improved accuracy from a 72.79% baseline to 76.02%.
Fine-tuning and Model Soups
In the context of "Model Soups" (averaging multiple fine-tuned models), TWA-v matches the performance of state-of-the-art "Greedy Soup" methods but requires 4x fewer models, demonstrating its efficiency in extracting value from a limited number of checkpoints.
Conclusion
Trainable Weight Averaging (TWA) provides a robust framework for adaptive weight averaging through subspace training. By moving beyond fixed heuristics to learned coefficients, TWA offers a significant boost in both training efficiency and model generalization. Future work will explore merging weights from disparate training trajectories and integrating TWA with other parameter-efficient fine-tuning methods like LoRA.
Summary updated