Training CNNs With Normalized Kernels

Mete Özay, Takayuki OkataniAAAI 2018 · OpenReview

Abstract

Several methods of normalizing convolution kernels have been proposed in the literature to train convolutional neural networks (CNNs), and have shown some success. However, our understanding of these methods has lagged behind their success in application; there are a lot of open questions, such as why a certain type of kernel normalization is effective and what type of normalization should be employed for each layer of a CNN. As the first step towards answering these questions, we propose a framework that enables us to use a variety of kernel normalization methods at any layer of a CNN. We pose estimation of convolution kernels under normalization constraints as constraint-free optimization on kernel submanifolds. To this end, we propose an algorithm for optimization on kernel manifolds in CNNs by appropriate scaling of the space of kernels based on structure of CNNs and statistics of data. Our experimental results show that the proposed method can successfully train popular CNN models using several different types of kernel normalization methods, improving classification performance and providing state-of-the-art performance for major image classification benchmarks.

Introduction

Training deep Convolutional Neural Networks (CNNs) is often hindered by the vanishing or exploding gradient problem. While kernel normalization techniques like Weight Normalization have been introduced to mitigate these issues, they are frequently applied in an ad-hoc manner. Standard optimization methods like Stochastic Gradient Descent (SGD) can become unstable when forced to maintain normalization constraints. This paper addresses these challenges by reformulating kernel estimation as an optimization problem on Riemannian submanifolds.

Methodology: Optimization on Kernel Manifolds

The core innovation lies in treating convolution kernels not as elements of a flat Euclidean space, but as points on specific geometric manifolds defined by normalization constraints:

  • Sphere Manifold: Constrains the unit Frobenius norm of the kernel.
  • Oblique Manifold: Constrains each column of the kernel matrix to have a unit norm.
  • Stiefel Manifold: Constrains kernels to be orthonormal, ensuring columns are perpendicular and of unit length.

Riemannian SGD and Scaling

To adapt manifold optimization for the hierarchical structure of CNNs, the authors propose a specialized Riemannian SGD algorithm. A critical component is the scaling factor $\Gamma_l^t$, which ensures stability across layers:

  1. Geometric Scaling ($\gamma_l$): Accounts for the layer's receptive field and channel dimensions.
  2. Statistical Scaling ($\lambda_l^t$): Dynamically adjusts based on the standard deviation of data activations.
Geometric illustration of the proposed manifold optimization process, showing the projection of the Euclidean gradient onto the tangent space and the subsequent retraction back to the manifold.

The optimization loop involves calculating the Euclidean gradient, projecting it onto the manifold's tangent space, moving along this surface, and applying a retraction to map the result back onto the manifold.

Experimental Results

The framework was evaluated on CIFAR-10, CIFAR-100, and ImageNet using architectures such as Network-in-Network (NiN) and ResNet.

Performance on CIFAR Benchmarks

The proposed method demonstrates significant improvements over baseline models and existing normalization techniques. For instance, on CIFAR-10 without data augmentation, the error rate for a NiN model was reduced from 10.68% to 8.57% using Stiefel manifold constraints. When combined with Mean-Only Batch Normalization (MOBN), the performance reached state-of-the-art levels.

Comparison of classification error rates on CIFAR-10 and CIFAR-100 with and without data augmentation (DA). The proposed MOBN variants (Sp/Ob/St) consistently outperform baselines.

Large-Scale Evaluation on ImageNet

On the ImageNet dataset, a ResNet-18 model trained with the proposed manifold optimization achieved a 28.14% Top-1 error rate, surpassing more complex models like Inception trained with PRONG. This highlights the scalability and effectiveness of the geometric approach in large-scale settings.

Conclusion

By leveraging the geometric properties of kernel manifolds and introducing a novel scaling mechanism, this work provides a theoretically grounded and practically effective framework for training CNNs. The results indicate that manifold optimization not only stabilizes training but also enhances the representational power of the network, leading to superior classification performance across diverse benchmarks. Future research may extend this framework to other architectures like Recurrent Neural Networks and tasks such as object detection.

Summary updated

Abstract

Several methods of normalizing convolution kernels have been proposed in the literature to train convolutional neural networks (CNNs), and have shown some success. However, our understanding of these methods has lagged behind their success in application; there are a lot of open questions, such as why a certain type of kernel normalization is effective and what type of normalization should be employed for each (e.g., higher or lower) layer of a CNN. As the first step towards answering these questions, we propose a framework that enables us to use a variety of kernel normalization methods at any layer of a CNN. A naive integration of kernel normalization with a general optimization method, such as SGD, often entails instability while updating parameters. Thus, existing methods employ ad-hoc procedures to empirically assure convergence. In this study, we pose estimation of convolution kernels under normalization constraints as constraint-free optimization on kernel submanifolds that are identified by the employed constraints. Note that naive application of the established optimization methods for matrix manifolds to the aforementioned problems is not feasible because of the hierarchical nature of CNNs. To this end, we propose an algorithm for optimization on kernel manifolds in CNNs by appropriate scaling of the space of kernels based on structure of CNNs and statistics of data. We theoretically prove that the proposed algorithm has assurance of almost sure convergence to a solution at single minimum. Our experimental results show that the proposed method can successfully train popular CNN models using several different types of kernel normalization methods. Moreover, they show that the proposed method improves classification performance of baseline CNNs, and provides state-of-the-art performance for major image classification benchmarks.
BibTeX
@inproceedings{ozay2018training,
  author = {Mete Özay and Takayuki Okatani},
  title = {{Training CNNs With Normalized Kernels}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=BkEH21-O-r},
}
Figure