Optimization on Product Submanifolds of Convolution Kernels

Mete Ozay, Takayuki OkataniarXiv 2017 · OpenReview

Abstract

Recent advances in training convolutional neural networks (CNNs) have shown that normalizing kernels according to specific geometric constraints can significantly improve performance. This work introduces a framework for training CNNs using ensembles of joint spaces of kernels, termed Ensembles of Products of Submanifolds (PEMs). We propose three strategies for constructing these ensembles and develop a geometry-aware SGD (G-SGD) algorithm that adaptively adjusts step sizes based on the curvature of these product spaces. Experimental results on CIFAR and ImageNet benchmarks demonstrate that G-SGD on PEMs improves training loss, convergence, and classification accuracy compared to standard Euclidean and single-manifold approaches.

Introduction

In standard deep learning, network weights are typically treated as points in a flat Euclidean space. However, forcing weights to reside on specific geometric shapes, or Riemannian manifolds (such as spheres or Stiefel manifolds), can enhance training stability and generalization. While previous methods often apply a uniform constraint across all kernels, this paper explores the benefits of using a mixture of different constraints. By combining multiple manifolds into a "product space," we can capture a richer set of features. However, these spaces possess complex curvatures that challenge standard optimization techniques. We address this by introducing Ensembles of Products of Submanifolds (PEMs) and a specialized geometry-aware optimizer.

Methodology

PEM Construction Strategies

We define three primary strategies for grouping kernels into ensembles of product submanifolds within a CNN layer:

  1. PI (Input channels): Kernels are grouped based on the input channel they process.
  2. PO (Output channels): Kernels are grouped according to the output feature map they produce.
  3. PIO (Input and Output): A hybrid approach that considers both input and output dimensions.

Each group within an ensemble can be assigned a distinct geometric constraint, such as a Sphere, Oblique, or Stiefel manifold, allowing for a heterogeneous geometric regularization of the network.

Strategies for constructing ensembles of PEMs in CNNs: (i) PI, (ii) PO, and (iii) PIO.

Geometry-Aware SGD (G-SGD)

To optimize over these complex spaces, we propose G-SGD. Unlike standard SGD, G-SGD operates directly on the manifolds. Its key innovation is an adaptive step size function that incorporates the sectional curvature and geodesic distance of the specific product manifold. This curvature-aware adjustment prevents the optimizer from overshooting in highly curved regions, ensuring stable convergence and preventing gradient-related issues common in deep manifold optimization.

Experimental Results

The proposed framework was evaluated using Residual Network (ResNet) architectures on CIFAR-10, CIFAR-100, and ImageNet (ILSVRC) datasets.

Performance Benchmarks

The G-SGD algorithm on PEMs consistently outperformed baseline Euclidean CNNs and single-manifold methods across all tested datasets.

  • CIFAR-10: Using a ResNet-44 architecture, the PIO strategy with a mixture of manifolds (Sphere + Oblique + Stiefel + Euclidean) achieved a 5.92% error rate, a notable improvement over the 7.05% Euclidean baseline.
  • ImageNet: On ResNet-18, the PIO strategy reached a 27.07% Top-1 error, compared to 30.31% for the baseline.
  • Scaling and Depth: The benefits of PEMs were particularly evident in deeper networks. For instance, on CIFAR-100 with a 110-layer ResNet (RCD), the error rate dropped from 27.01% to 23.79% when using the PIO-PEM approach.
Comparative performance of various models on CIFAR-10 and CIFAR-100 datasets with and without data augmentation (DA).

Key Insights

  • Geometric Diversity: The best-performing models utilized a combination of multiple manifold types, suggesting that different parts of a network benefit from varied geometric regularizers.
  • Curvature-Driven Regularization: The sectional curvature of the product manifolds provides a natural mechanism for regulating learning rates, leading to smoother loss reduction.
  • Cohesive Learning: By optimizing kernels as part of a product manifold, the updates implicitly account for inter-kernel relationships, resulting in more effective feature extraction.

Conclusion

This work demonstrates that training CNNs on Ensembles of Products of Submanifolds (PEMs) using a geometry-aware optimizer can significantly boost performance and convergence. The proposed PIO strategy and G-SGD algorithm provide a robust framework for leveraging complex geometric constraints in deep learning. Future work will explore the impact of overlapping PEM sets and the application of this framework to other architectures like Recurrent Neural Networks.

Summary updated

Abstract

Recent advances in optimization methods used for training convolutional neural networks (CNNs) with kernels, which are normalized according to particular constraints, have shown remarkable success. This work introduces an approach for training CNNs using ensembles of joint spaces of kernels constructed using different constraints. For this purpose, we address a problem of optimization on ensembles of products of submanifolds (PEMs) of convolution kernels. To this end, we first propose three strategies to construct ensembles of PEMs in CNNs. Next, we expound their geometric properties (metric and curvature properties) in CNNs. We make use of our theoretical results by developing a geometry-aware SGD algorithm (G-SGD) for optimization on ensembles of PEMs to train CNNs. Moreover, we analyze convergence properties of G-SGD considering geometric properties of PEMs. In the experimental analyses, we employ G-SGD to train CNNs on Cifar-10, Cifar-100 and Imagenet datasets. The results show that geometric adaptive step size computation methods of G-SGD can improve training loss and convergence properties of CNNs. Moreover, we observe that classification performance of baseline CNNs can be boosted using G-SGD on ensembles of PEMs identified by multiple constraints.
BibTeX
@misc{ozay2017optimization,
  author = {Mete Ozay and Takayuki Okatani},
  title = {{Optimization on Product Submanifolds of Convolution Kernels}},
  year = {2017},
  eprint = {1701.06123},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1701.06123},
}
Figure