Abstract
In this paper, we propose a novel approach to enhance the generalization performance of deep neural networks. Our method employs a hierarchical hypersphere-based constraint that organizes weight vectors hierarchically based on observed data. By diversifying the parameter space of hyperplanes in the classification layer, we aim to encourage discriminative generalization. We introduce a self-supervised grouping method designed to unveil hierarchical structures in scenarios with unknown hierarchy information. To maximize distances between weight vectors on multiple hyperspheres, we propose a novel metric that combines discrete and continuous measures. This regularization encourages diverse orientations, consequently leading to improved generalization. Extensive evaluations on datasets, including CUB200-2011, Stanford-Cars, CIFAR-100, and TinyImageNet, consistently demonstrate enhancements in classification performance compared to baseline settings.
Introduction
Deep neural networks often encounter a significant "generalization gap," where high performance on training datasets does not translate to unseen test data. A promising solution is diversity-promoting learning, which encourages weight vectors—the hyperplanes defining class boundaries—to be as distinct as possible. However, existing methods often suffer from high computational complexity or ignore the inherent semantic structure of the data. For instance, treating a "Beagle" and a "Bulldog" as equally distinct as a "Beagle" and a "Car" ignores useful inductive biases. This research addresses these flaws by introducing a hierarchical hypersphere-based regularization technique.
Methodology
The proposed method reparameterizes weights to exist on multiple, nested spheres, turning high-dimensional optimization into a structured geometric problem.
Hierarchical Parameterization
Instead of a single weight vector, each weight is defined as the difference between a "surface" parameter and a "center" parameter ($w = w_{surface} - w_{center}$). This allows groups of semantically related classes (superclasses) to share a center on one hypersphere while their individual subclasses diverge on its surface, as shown in the geometric visualization below.
Self-Supervised Grouping
To handle datasets without pre-defined hierarchies, the authors developed a self-supervised algorithm that analyzes the model's "confusion probability." If the network frequently confuses two classes, they are automatically grouped into a superclass. This functional grouping often identifies more useful relationships for regularization than traditional taxonomies.
Blended Metrics for Diversity
To maximize diversity, the authors introduce "blended" metrics using Pythagorean means (Arithmetic, Geometric, and Harmonic). These metrics combine continuous angular distance with a discrete, sign-based metric (similar to Hamming distance). This ensures that weights are not only separated by degrees but also occupy different quadrants of the parameter space, leading to more robust and discriminative features.
Experimental Evaluation
The method was evaluated on four standard computer vision benchmarks: CUB200-2011, Stanford Cars, CIFAR-100, and TinyImageNet. The results consistently showed that the proposed hierarchical regularization outperforms standard ResNet baselines and other state-of-the-art diversity metrics.
- CUB200-2011 (ResNet-18): Accuracy improved from a baseline of 71.95% to 77.02% using the blended arithmetic mean metric.
- CIFAR-100: Accuracy increased from 70.00% to 70.97%.
- TinyImageNet: Performance rose from 64.54% to 65.89%.
The "Blended" metrics consistently occupied the top performance spots across all datasets, validating the strategy of combining discrete and continuous distance measures.
Conclusion
By confining weights to hierarchical hyperspheres and utilizing self-supervised grouping, this method provides a powerful geometric regularization framework. It effectively captures semantic relationships and promotes parameter diversity, leading to significant gains in generalization. Future work may explore deeper multi-level hierarchies and integration with non-Euclidean manifolds, such as hyperbolic embeddings, to further enhance hierarchical data representation.
Summary updated