BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Training

Wenjie Zhou, Bohan Wang, Wei Chen, Xueqi Cheng

Abstract

Deep learning optimization exhibits a fundamental dichotomy: parameter updates along the top eigendirections of the loss Hessian (dominant subspace) capture most of the magnitude but contribute minimally to loss reduction, while updates in the orthogonal component (bulk subspace) drive most learning progress. We introduce the Bulk-Space-Filtration-Accelerator (BSFA), a plug-and-play framework that accelerates training by differentially scaling these components. BSFA moderates updates in the dominant subspace to enhance stability and amplifies those in the bulk subspace to boost convergence. Using an efficient PCA-based estimator and a block-wise strategy, BSFA is scalable to large models, achieving a 2x speedup in pre-training LLaMA models and significant gains in vision tasks.

The Subspace Dichotomy Problem

Training modern Large Language Models (LLMs) and Vision Transformers (ViTs) is computationally intensive. Recent research has identified the "subspace dichotomy" as a key factor in optimization efficiency. In this phenomenon, the majority of weight update magnitude occurs along a few "dominant" directions—the top eigenvectors of the loss Hessian. However, these large updates often result in "jittery" training and contribute little to actual loss reduction. Conversely, the "bulk" of the remaining directions, despite having smaller update magnitudes, is where the primary learning occurs.

Figure 1: Demonstration of the subspace dichotomy on SST2. Updates projected onto the bulk subspace drive the majority of loss reduction compared to the dominant subspace.

Exploiting this dichotomy traditionally requires computing the Hessian matrix, which is computationally prohibitive for large-scale models.

The BSFA Framework

BSFA addresses this by acting as a wrapper around standard optimizers like AdamW or SGD. It employs three core innovations:

  1. Differential Scaling: BSFA scales updates in the dominant subspace by a factor $\alpha$ (typically $< 1$) to improve stability and updates in the bulk subspace by a factor $\gamma$ (typically $> 1$) to accelerate convergence.
  2. PCA-based Projector Estimator (PPE): Instead of expensive second-order calculations, BSFA performs Principal Component Analysis (PCA) on a history of recent gradients. The directions with the highest variance in gradient history correspond to the dominant directions of the Hessian. This PPE approach is nearly 100x faster than traditional methods like Lanczos.
  3. Block-wise Strategy (BPPE): To maintain scalability, the PCA estimation is applied to individual parameter blocks (e.g., layers) rather than the entire model. This leverages the block-diagonal structure of the Hessian and keeps memory overhead manageable.

Key Results and Efficiency

BSFA demonstrates substantial acceleration across various architectures and datasets. On LLaMA-72M and LLaMA-134M, BSFA reached baseline loss levels in approximately half the training steps. In computer vision tasks, the gains were even more pronounced.

Figure 6: Performance of BSFA on CIFAR-100 using DenseNet121. BSFA achieves higher test accuracy and lower validation loss in 50 epochs than the baseline does in 200 epochs.

As shown in Figure 6, BSFA can achieve a 4x speedup on CNN architectures, reaching superior performance in 50 epochs compared to a 200-epoch baseline.

Memory and Stability

While storing gradient history introduces memory overhead, the authors found that 4-bit quantization can reduce this requirement by roughly 80% without degrading training quality. Furthermore, ablation studies highlighted that applying BSFA to normalization layers is essential; omitting them often leads to training divergence, emphasizing the role of the dominant subspace in maintaining stability.

Conclusion and Future Directions

BSFA provides a practical, scalable method to exploit the geometric properties of the loss landscape. By decoupling the "brakes" (stability in the dominant subspace) from the "gas pedal" (progress in the bulk subspace), it offers a significant leap in training efficiency. Future work may explore adaptive strategies for assigning principal components to different layers and further optimizing the memory footprint for models exceeding 100 billion parameters.

Summary updated

Abstract

highlight a fundamental dichotomy in deep learning optimization: Although parameter updates along the top eigendirections of the loss Hessian (Domspace) capture most of the update magnitude, they often contribute minimally to loss reduction.In contrast, updates in the orthogonal component (Bulk-space) have smaller magnitudes but drive most learning progress.In this work, we further advance the understanding of this phenomenon and introduce the Bulk-Space-Filtration-Accelerator (BSFA), a novel plugand-play framework.BSFA accelerates training by differentially scaling update components projected onto these distinct subspaces, simultaneously enhancing stability by moderating updates in the dominant subspace and boosting convergence speed by amplifying those in the bulk-space.To ensure BSFA is both practical and scalable for contemporary large models, we introduce two key innovations: an efficient estimator using Principal Component Analysis (PCA) on historical updates for fast subspace estimation, and a block-wise strategy that applies this estimation on a per-parameter-block basis.These designs make BSFA computationally tractable and highly effective.We demonstrate BSFA's acceleration across various tasks, notably achieving approximately 2 speedup when pre-training LLaMA-72M on WikiText-103 and LLaMA-134M on OpenWebText compared to vanilla AdamW.
BibTeX
@misc{zhou2025bsfa,
  author = {Wenjie Zhou and Bohan Wang and Wei Chen and Xueqi Cheng},
  title = {{BSFA: Leveraging the Subspace Dichotomy to Accelerate Neural Network Training}},
  year = {2025},
  eprint = {2510.25244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.25244},
}
Figure