Visual Similarity from Optimizing Feature and Memory On A Hypersphere

Xinlei Pan, Rudrasis Chakraborty, Stella X. YuWorkshops @ CVPR 2019 · OpenReview

Abstract

Supervised learning for image classification relies on extensive human annotation to develop latent feature representations. We present an unsupervised metric learning approach that derives visual similarity directly from raw images. By mapping high-dimensional data onto a low-dimensional hypersphere and optimizing feature representations within a visual memory bank, our method achieves maximal discrimination among instances. Our formulation, rooted in Riemannian geometry, outperforms existing instance discrimination algorithms in classification accuracy, convergence rate, and feature transferability. Furthermore, we demonstrate the utility of these learned features in improving sample efficiency for reinforcement learning tasks.

Introduction

The success of deep learning in computer vision is largely attributed to large-scale supervised datasets. However, the cost of annotation motivates the search for effective unsupervised alternatives. A prominent direction is instance-wise discriminative learning, which treats every image as a unique class. While previous methods often project features onto a unit sphere, they frequently rely on Euclidean metrics for optimization. We argue that because the feature space is a curved manifold, the optimization should respect its natural geometry.

Methodology: Riemannian Optimization on a Hypersphere

Our approach treats the feature space as a Riemannian manifold. We utilize a ResNet-50 architecture to map input images to points on a unit hypersphere ($S^m$). To manage the large number of instances, we maintain a non-parametric visual memory bank that stores consolidated representations.

The core innovation lies in the use of arc-cosine distance (geodesic distance) rather than standard cosine similarity or Euclidean distance. By employing Riemannian Stochastic Gradient Descent (SGD), we ensure that updates to the memory bank remain on the hypersphere's surface. This geometric consistency allows the model to better capture the "natural" visual similarities between images.

Implicit Clustering and Feature Separation

A key benefit of maximizing instance discrimination on a hypersphere is the emergence of implicit semantic clusters. Even without explicit labels, visually similar objects naturally group together as the optimization process pushes distinctive instances apart on the manifold.

Comparison of feature embeddings between our Riemannian approach (left) and the Euclidean baseline (right). Our method shows significantly clearer cluster separation.

As shown in the t-SNE visualizations above, our method produces more distinct and cohesive clusters compared to the baseline, which exhibits significant mixing between different semantic categories.

Performance and Convergence

We evaluated our framework on benchmarks including MNIST, CIFAR-10, and SVHN. Our method consistently outperforms the state-of-the-art baseline (Wu et al., 2018). For instance, on CIFAR-10, we achieved an accuracy of 82.53% compared to the baseline's 80.65%.

Testing accuracy vs. number of training epochs on CIFAR-10. Our method (orange) demonstrates faster convergence and higher final accuracy than the baseline (blue).

Beyond accuracy, the convergence rate is notably improved. As illustrated in the figure above, our Riemannian approach reaches peak performance in fewer epochs, indicating a more efficient optimization path.

Transferability and Reinforcement Learning

The features learned through our unsupervised process are highly transferable. When pre-trained on MNIST and tested on SVHN, our model outperformed the baseline, suggesting that the hypersphere-based features capture more fundamental visual structures.

Furthermore, we applied these features to vision-based reinforcement learning (RL) in environments like TORCS and Atari. By using our pre-trained feature extractor, RL agents achieved higher rewards with fewer samples, demonstrating that unsupervised metric learning can significantly boost sample efficiency in complex control tasks.

Conclusion

By aligning the optimization process with the underlying geometry of the feature space, we provide a more principled and effective framework for unsupervised learning. Our results show that respecting the hypersphere manifold leads to superior feature representations that are both discriminative and transferable. Future work will focus on scaling this approach to massive datasets like ImageNet and exploring more complex manifold structures.

Summary updated

Abstract

Supervised learning of classification from annotated images develops a latent feature representation that captures semantic visual similarity. We propose an unsupervised metric learning method that develops apparent visual similarity from images alone. Our method maps high-dimensional visual data onto a low-dimensional hyper-sphere and consolidate such feature representations into a visual memory representation. Optimizing the feature mapping and visual memory on a hypersphere achieves maximal discrimination among instances. Our formulation and solution is not only more principled in theory than closely related unsupervised instance discrimination algorithms, but also better in practice in terms of classification accuracy, convergence rate, and feature transferability. We also show that our learned feature can be very useful for vision-based reinforcement learning tasks to improve sample efficiency.
BibTeX
@inproceedings{pan2019visual,
  author = {Xinlei Pan and Rudrasis Chakraborty and Stella X. Yu},
  title = {{Visual Similarity from Optimizing Feature and Memory On A Hypersphere}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=sBh3bT65-K3},
}
Figure