Trainable Weight Averaging for Fast Convergence and Better Generalization

Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, Xiaolin Huang

Abstract

Training deep neural networks (DNNs) in low-dimensional subspaces is a promising direction for achieving efficient training and better generalization performance. Previous works extract the subspaces by using random projection or performing dimensionality reduction method on the training trajectory, but these methods can be inefficient or unstable in terms of dimensionality and numerical operations. In this paper, we connect subspace training to weight averaging and propose Trainable Weight Averaging (TWA), a general approach for subspace training that generalizes the previous efforts. TWA is efficient in terms of dimensionality and also easy to use, making it a promising new method for subspace training. We further design an efficient scheme for subspace training to cope with large-scale problems, which allows parallel training across multiple nodes and evenly distributing the memory and computation burden to each node. We apply TWA to efficient neural network training and improving fine-tuning performance tasks to demonstrate the great efficiency and effectiveness of our approach. We conduct extensive experiments that cover various benchmark computer vision and neural language processing tasks with various architectures.

Introduction

Weight averaging has become a standard technique in deep learning to improve model generalization by smoothing the loss landscape. Methods like Stochastic Weight Averaging (SWA) and Exponential Moving Averages (EMA) are widely used but rely on fixed or heuristic weighting schemes. For instance, SWA typically assigns equal weight to all checkpoints in a trajectory. However, these fixed schemes are often suboptimal because checkpoints at different stages of training contribute differently to the final performance.

To address these limitations, we introduce Trainable Weight Averaging (TWA). TWA transforms weight averaging into an optimization problem within a low-dimensional subspace. By learning the optimal coefficients for combining model checkpoints, TWA achieves faster convergence and superior generalization compared to traditional fixed-weight methods.

Methodology

The core of TWA is the conceptualization of weight averaging as a subspace training task. Instead of optimizing in the high-dimensional full-parameter space $\mathbb{R}^D$, TWA operates in a tiny subspace $\mathbb{R}^n$ spanned by $n$ candidate weight checkpoints.

Subspace Construction and Optimization

TWA constructs a set of basis vectors from historical model checkpoints. To ensure stability and speed, these checkpoints are decoupled using decentralization and normalization, a process significantly more efficient than traditional Gram-Schmidt orthogonalization.

Summary of notations used in the TWA framework.

We propose two primary variants:

  1. TWA-t: Optimizes coefficients using the training dataset.
  2. TWA-v: Uses a small, held-out validation set (e.g., 2% of training data). TWA-v acts as a powerful regularizer, finding the combination of weights that generalizes best to unseen data.
Comparison of TWA with existing weight averaging and ensemble methods.

As shown in the comparison above, TWA differs from methods like SWA or EMA by defining its solution set as the span of checkpoints rather than a fixed average. This allows for a higher-dimensional solution space ($n$ or $nL$ for layer-wise TWA) while maintaining $O(1)$ inference cost, unlike traditional ensembles which scale linearly with the number of models.

Efficiency and Scalability

To handle large-scale models, TWA incorporates several efficiency-focused designs:

  • Distributed Training: The projection matrix is partitioned across multiple GPUs to prevent memory bottlenecks.
  • Low-Bit Quantization: The projection matrix can be quantized to 4 bits with negligible performance loss, reducing memory usage by 8x.
  • Layer-wise Processing: TWA can learn independent coefficients for different network layers, providing finer control over the model's composition.

Experimental Results

We evaluated TWA across diverse tasks, including image classification (CIFAR, ImageNet), machine translation (WMT2014), and language modeling (WikiText-2), using architectures ranging from ResNets to Transformers.

Training Acceleration and Generalization

TWA significantly reduces training time. On CIFAR-100, we observed a 45% reduction in total training time. On ImageNet, TWA achieved a 32% reduction while maintaining or exceeding baseline accuracy. Notably, for ViT-B/16 on ImageNet, TWA-v improved accuracy from a 72.79% baseline to 76.02%.

Fine-tuning and Model Soups

In the context of "Model Soups" (averaging multiple fine-tuned models), TWA-v matches the performance of state-of-the-art "Greedy Soup" methods but requires 4x fewer models, demonstrating its efficiency in extracting value from a limited number of checkpoints.

Conclusion

Trainable Weight Averaging (TWA) provides a robust framework for adaptive weight averaging through subspace training. By moving beyond fixed heuristics to learned coefficients, TWA offers a significant boost in both training efficiency and model generalization. Future work will explore merging weights from disparate training trajectories and integrating TWA with other parameter-efficient fine-tuning methods like LoRA.

Summary updated

Abstract

Training deep neural networks (DNNs) in low-dimensional subspaces is a promising direction for achieving efficient training and better generalization performance. Previous works extract the subspaces by using random projection or performing dimensionality reduction method on the training trajectory, but these methods can be inefficient or unstable in terms of dimensionality and numerical operations. In this paper, we connect subspace training to weight averaging and propose Trainable Weight Averaging (TWA), a general approach for subspace training that generalizes the previous efforts. TWA is efficient in terms of dimensionality and also easy to use, making it a promising new method for subspace training. We further design an efficient scheme for subspace training to cope with large-scale problems, which allows parallel training across multiple nodes and evenly distributing the memory and computation burden to each node. We apply TWA to efficient neural network training and improving fine-tuning performance tasks to demonstrate the great efficiency and effectiveness of our approach. We conduct extensive experiments that cover various benchmark computer vision and neural language processing tasks with various architectures. The code of implementation is available at https://github.com/nblt/TWA.
BibTeX
@misc{li2022trainable,
  author = {Tao Li and Zhehao Huang and Qinghua Tao and Yingwen Wu and Xiaolin Huang},
  title = {{Trainable Weight Averaging for Fast Convergence and Better Generalization}},
  year = {2022},
  eprint = {2205.13104},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.13104},
}
Figure