Introduction
Modern neural networks are characterized by extreme over-parameterization, often containing far more weights than necessary to solve a given task. While this redundancy aids optimization, it imposes massive memory requirements on hardware. Dynamical Low-Rank Training (DLRT) has emerged as a promising solution, where weight matrices are factorized (e.g., $W \approx USV^\top$) and only these smaller factors are updated during training.
However, the mathematical integrators used to update these factors—specifically Projector Splitting Integrators (PSI)—face a significant hurdle in machine learning: the "backward-in-time" step. While effective for physics simulations, this step frequently causes training loss to spike in neural networks, undermining convergence. This paper introduces the Augmented Backward-Corrected Projector Splitting Integrator (abc-PSI), a robust, rank-adaptive method that guarantees loss descent and improves computational efficiency.
The abc-PSI Framework
The abc-PSI method addresses the stability and efficiency limitations of previous DLRT approaches through three core innovations:
- Backward Correction: The algorithm replaces the problematic backward-in-time step with a projection step that ensures all updates move forward in time, aligning the integrator with the unidirectional nature of gradient descent.
- Basis Augmentation: This is the central innovation of the paper. Before updating the weights, the algorithm "augments" the current mathematical basis with the basis from the previous step. This ensures that the new low-rank space always contains the previous solution, providing a mathematical guarantee that the training loss will decrease (or remain stable) at every iteration.
- Rank Adaptivity: Because the augmentation step doubles the rank, the algorithm employs a Singular Value Decomposition (SVD) with a truncation tolerance ($\tau$). This allows the network to dynamically prune unimportant dimensions, finding an optimal compressed size throughout the training process.
Computational Efficiency
Beyond stability, abc-PSI is designed for high performance. Most low-rank methods are bottlenecked by expensive mathematical operations like QR decompositions and SVDs. abc-PSI is computationally leaner than the popular "BUG" integrators, requiring only one QR decomposition and one SVD per iteration, whereas competing methods often require two QR steps. This reduction makes DLRT more competitive with standard "dense" training in terms of wall-clock time.
Experimental Evaluation
The researchers evaluated abc-PSI across different scales and architectures to demonstrate its versatility:
Stability on MNIST
Using a five-layer fully connected network trained from scratch, the authors tested the method's limits. abc-PSI remained stable and converged even at high compression rates (exceeding 90%) and high learning rates—scenarios where original PSI and standard backward-corrected PSI failed completely. In lower compression regimes, abc-PSI achieved an average test accuracy of 97.42%, outperforming its un-augmented counterparts.
Fine-Tuning Vision Transformers (ViT)
To test the method on modern architectures, the authors fine-tuned a pre-trained Vision Transformer (ViT-base-patch16-224) on Cifar10 and Cifar100. abc-PSI achieved accuracy comparable to industry-standard methods like LoRA and AdaLoRA but with a significantly smaller parameter footprint. For example, on Cifar10, abc-PSI utilized only 0.34M parameters compared to 0.47M for AdaLoRA.
Conclusion and Future Directions
abc-PSI provides a robust "safety net" for low-rank training by ensuring that the loss function behaves predictably. By combining this stability with a reduced computational overhead, the method makes DLRT a more viable candidate for memory-constrained environments.
While the current work focuses on fully connected and Transformer layers, future research may extend these benefits to convolutional layers or extremely large-scale language models (LLMs). The efficiency gains of abc-PSI are particularly promising for the future of on-device training and federated learning, where minimizing memory and compute is critical.
Summary updated