Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality
Lists
Only you see this
Abstract
Inserting an SVD meta-layer into neural networks is prone to<br> make the covariance ill-conditioned, which could harm the model in the<br> training stability and generalization abilities. In this paper, we systematically<br> study how to improve the covariance conditioning by enforcing<br> orthogonality to the Pre-SVD layer. Existing orthogonal treatments on<br> the weights are first investigated. However, these techniques can improve<br> the conditioning but would hurt the performance. To avoid such a side<br> effect, we propose the Nearest Orthogonal Gradient (NOG) and Optimal<br> Learning Rate (OLR). The effectiveness of our methods is validated<br> in two applications: decorrelated Batch Normalization (BN) and Global<br> Covariance Pooling (GCP). Extensive experiments on visual recognition<br> demonstrate that our methods can simultaneously improve the covariance<br> conditioning and generalization. Moreover, the combinations with<br> orthogonal weight can further boost the performances.
BibTeX
@inproceedings{yue2023improving,
author = {Song Yue and Nicu Sebe and Wei Wang},
title = {{Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality}},
booktitle = {Zenodo},
year = {2023},
doi = {10.5281/zenodo.7566129},
}Loading related content...