Abstract
Supervised learning for image classification relies on extensive human annotation to develop latent feature representations. We present an unsupervised metric learning approach that derives visual similarity directly from raw images. By mapping high-dimensional data onto a low-dimensional hypersphere and optimizing feature representations within a visual memory bank, our method achieves maximal discrimination among instances. Our formulation, rooted in Riemannian geometry, outperforms existing instance discrimination algorithms in classification accuracy, convergence rate, and feature transferability. Furthermore, we demonstrate the utility of these learned features in improving sample efficiency for reinforcement learning tasks.
Introduction
The success of deep learning in computer vision is largely attributed to large-scale supervised datasets. However, the cost of annotation motivates the search for effective unsupervised alternatives. A prominent direction is instance-wise discriminative learning, which treats every image as a unique class. While previous methods often project features onto a unit sphere, they frequently rely on Euclidean metrics for optimization. We argue that because the feature space is a curved manifold, the optimization should respect its natural geometry.
Methodology: Riemannian Optimization on a Hypersphere
Our approach treats the feature space as a Riemannian manifold. We utilize a ResNet-50 architecture to map input images to points on a unit hypersphere ($S^m$). To manage the large number of instances, we maintain a non-parametric visual memory bank that stores consolidated representations.
The core innovation lies in the use of arc-cosine distance (geodesic distance) rather than standard cosine similarity or Euclidean distance. By employing Riemannian Stochastic Gradient Descent (SGD), we ensure that updates to the memory bank remain on the hypersphere's surface. This geometric consistency allows the model to better capture the "natural" visual similarities between images.
Implicit Clustering and Feature Separation
A key benefit of maximizing instance discrimination on a hypersphere is the emergence of implicit semantic clusters. Even without explicit labels, visually similar objects naturally group together as the optimization process pushes distinctive instances apart on the manifold.
As shown in the t-SNE visualizations above, our method produces more distinct and cohesive clusters compared to the baseline, which exhibits significant mixing between different semantic categories.
Performance and Convergence
We evaluated our framework on benchmarks including MNIST, CIFAR-10, and SVHN. Our method consistently outperforms the state-of-the-art baseline (Wu et al., 2018). For instance, on CIFAR-10, we achieved an accuracy of 82.53% compared to the baseline's 80.65%.
Beyond accuracy, the convergence rate is notably improved. As illustrated in the figure above, our Riemannian approach reaches peak performance in fewer epochs, indicating a more efficient optimization path.
Transferability and Reinforcement Learning
The features learned through our unsupervised process are highly transferable. When pre-trained on MNIST and tested on SVHN, our model outperformed the baseline, suggesting that the hypersphere-based features capture more fundamental visual structures.
Furthermore, we applied these features to vision-based reinforcement learning (RL) in environments like TORCS and Atari. By using our pre-trained feature extractor, RL agents achieved higher rewards with fewer samples, demonstrating that unsupervised metric learning can significantly boost sample efficiency in complex control tasks.
Conclusion
By aligning the optimization process with the underlying geometry of the feature space, we provide a more principled and effective framework for unsupervised learning. Our results show that respecting the hypersphere manifold leads to superior feature representations that are both discriminative and transferable. Future work will focus on scaling this approach to massive datasets like ImageNet and exploring more complex manifold structures.
Summary updated