Group Orthogonalization Regularization for Vision Models Adaptation and Robustness

Yoav Kurtz, Noga Bar, Raja Giryes

Introduction

As deep neural networks increase in depth and width, they frequently encounter the problem of parameter redundancy. In high-dimensional models, convolutional filters within the same layer often become highly correlated, which restricts the model's expressivity—its capacity to represent a diverse set of features. While enforcing global orthogonality (ensuring all filters are perpendicular and have a unit norm) can mitigate this, the computational cost is prohibitive. The complexity of global orthogonalization scales quadratically with the number of filters, creating a significant bottleneck for modern architectures.

This paper introduces Group Orthogonalization Regularization (GOR), a computationally efficient "orthogonality-lite" approach. GOR seeks a middle ground by encouraging groups of filters to be orthonormal, thereby reaping the benefits of feature diversity and improved generalization without the massive computational tax associated with full layer-wise orthogonalization.

Methodology: Group Orthogonalization Regularization

GOR operates by partitioning filters into $N$ smaller groups rather than treating the entire layer as a single block. The regularization is then applied locally within these partitions.

The GOR Process

  1. Grouping: Filters are divided into $N$ distinct groups.
  2. Local Orthonormality: A regularization term, based on the Frobenius norm of the Gram matrix minus the identity matrix, is applied within these groups. This encourages the filters within a group to be mutually orthogonal and have unit length.
  3. Efficiency Gains: By restricting the calculation to smaller groups, the computational complexity is reduced from $O(C_{out}^2 C_{in})$ to $O(C_{out}^2 C_{in} / N)$, where $C_{out}$ and $C_{in}$ represent the number of output and input channels, respectively.

Partitioning Strategies

The authors explore two primary strategies for applying GOR:

  • Inter-Group: This is the primary focus of the research, where orthonormality is enforced within each group to ensure local feature diversity.
  • Intra-Group: This strategy enforces orthonormality between different groups. This is particularly effective when the model utilizes Group Normalization (GN), as it helps decorrelate the features that GN aims to normalize.

GOR is designed to be modular, integrating seamlessly into low-rank adaptation methods such as LoRA (for diffusion models) and AdaptMLP (for Vision Transformers). In these contexts, GOR is applied to the "up-projection" layers, where maintaining parameter expressivity is most critical for performance.

Experimental Evaluation

The effectiveness of GOR was evaluated across image classification, generative modeling, and adversarial robustness benchmarks.

Adaptation of Vision Transformers (ViTs)

In experiments involving ViT adaptation using AdaptFormer, GOR consistently outperformed standard fine-tuning methods. On the CIFAR-100 dataset, the inclusion of GOR improved accuracy from 85.9% to 86.16% when using a self-supervised backbone, and from 91.86% to 92.49% with a supervised backbone. These results suggest that GOR prevents the few parameters updated during adaptation from becoming redundant.

Generative Quality in Diffusion Models

When fine-tuning diffusion models on datasets like Oxford102, Pokemon-BLIP, and FS-COCO, GOR significantly improved image fidelity. This was measured by a consistent reduction in the Fréchet Inception Distance (FID) across all tested datasets. Qualitative analysis indicated that models regularized with GOR produced sharper details and demonstrated better alignment with text prompts compared to standard LoRA fine-tuning.

Adversarial Robustness

GOR also contributes to model security. When integrated into adversarial training frameworks like TRADES or Fast Adversarial Training (FAT), GOR improved both "Natural" accuracy and robustness against "AutoAttack" scenarios. For instance, in a setup using TRADES and Group Normalization, GOR increased AutoAttack accuracy from 49.72% to 51.56%. The authors attribute this to the model learning a more diverse set of robust features that are more difficult for adversarial perturbations to exploit.

Computational Efficiency

A key advantage of GOR is its efficiency. Compared to full Soft-Orthogonalization (SO), GOR significantly reduces the number of Multiply-Accumulate (MAC) operations. This allows researchers to apply regularization to wider layers and more complex architectures where full orthogonalization would be computationally infeasible.

Conclusion and Limitations

Group Orthogonalization Regularization provides a "sweet spot" for neural network regularization. By enforcing local diversity through grouped filters, it enhances the performance of both discriminative and generative models while maintaining high computational efficiency.

However, the method introduces two hyperparameters: the regularization strength ($\lambda$) and the number of groups ($N$). The optimal values for these parameters can vary depending on the architecture and the specific task. While GOR is highly effective in models using Group Normalization, finding the optimal grouping structure for models using other normalization techniques may require additional empirical search. Future work may focus on automating the selection of these hyperparameters or identifying specific layers that benefit most from GOR's diversity pressure.

Summary updated

BibTeX
@inproceedings{kurtz2023group,
  author = {Yoav Kurtz and Noga Bar and Raja Giryes},
  title = {{Group Orthogonalization Regularization for Vision Models Adaptation and Robustness}},
  booktitle = {BMVC},
  year = {2023},
}
Figure