Abstract
Recent advances in training convolutional neural networks (CNNs) have shown that normalizing kernels according to specific geometric constraints can significantly improve performance. This work introduces a framework for training CNNs using ensembles of joint spaces of kernels, termed Ensembles of Products of Submanifolds (PEMs). We propose three strategies for constructing these ensembles and develop a geometry-aware SGD (G-SGD) algorithm that adaptively adjusts step sizes based on the curvature of these product spaces. Experimental results on CIFAR and ImageNet benchmarks demonstrate that G-SGD on PEMs improves training loss, convergence, and classification accuracy compared to standard Euclidean and single-manifold approaches.
Introduction
In standard deep learning, network weights are typically treated as points in a flat Euclidean space. However, forcing weights to reside on specific geometric shapes, or Riemannian manifolds (such as spheres or Stiefel manifolds), can enhance training stability and generalization. While previous methods often apply a uniform constraint across all kernels, this paper explores the benefits of using a mixture of different constraints. By combining multiple manifolds into a "product space," we can capture a richer set of features. However, these spaces possess complex curvatures that challenge standard optimization techniques. We address this by introducing Ensembles of Products of Submanifolds (PEMs) and a specialized geometry-aware optimizer.
Methodology
PEM Construction Strategies
We define three primary strategies for grouping kernels into ensembles of product submanifolds within a CNN layer:
- PI (Input channels): Kernels are grouped based on the input channel they process.
- PO (Output channels): Kernels are grouped according to the output feature map they produce.
- PIO (Input and Output): A hybrid approach that considers both input and output dimensions.
Each group within an ensemble can be assigned a distinct geometric constraint, such as a Sphere, Oblique, or Stiefel manifold, allowing for a heterogeneous geometric regularization of the network.
Geometry-Aware SGD (G-SGD)
To optimize over these complex spaces, we propose G-SGD. Unlike standard SGD, G-SGD operates directly on the manifolds. Its key innovation is an adaptive step size function that incorporates the sectional curvature and geodesic distance of the specific product manifold. This curvature-aware adjustment prevents the optimizer from overshooting in highly curved regions, ensuring stable convergence and preventing gradient-related issues common in deep manifold optimization.
Experimental Results
The proposed framework was evaluated using Residual Network (ResNet) architectures on CIFAR-10, CIFAR-100, and ImageNet (ILSVRC) datasets.
Performance Benchmarks
The G-SGD algorithm on PEMs consistently outperformed baseline Euclidean CNNs and single-manifold methods across all tested datasets.
- CIFAR-10: Using a ResNet-44 architecture, the PIO strategy with a mixture of manifolds (Sphere + Oblique + Stiefel + Euclidean) achieved a 5.92% error rate, a notable improvement over the 7.05% Euclidean baseline.
- ImageNet: On ResNet-18, the PIO strategy reached a 27.07% Top-1 error, compared to 30.31% for the baseline.
- Scaling and Depth: The benefits of PEMs were particularly evident in deeper networks. For instance, on CIFAR-100 with a 110-layer ResNet (RCD), the error rate dropped from 27.01% to 23.79% when using the PIO-PEM approach.
Key Insights
- Geometric Diversity: The best-performing models utilized a combination of multiple manifold types, suggesting that different parts of a network benefit from varied geometric regularizers.
- Curvature-Driven Regularization: The sectional curvature of the product manifolds provides a natural mechanism for regulating learning rates, leading to smoother loss reduction.
- Cohesive Learning: By optimizing kernels as part of a product manifold, the updates implicitly account for inter-kernel relationships, resulting in more effective feature extraction.
Conclusion
This work demonstrates that training CNNs on Ensembles of Products of Submanifolds (PEMs) using a geometry-aware optimizer can significantly boost performance and convergence. The proposed PIO strategy and G-SGD algorithm provide a robust framework for leveraging complex geometric constraints in deep learning. Future work will explore the impact of overlapping PEM sets and the application of this framework to other architectures like Recurrent Neural Networks.
Summary updated