Jointly Learning Multiple Measures of Similarities from Triplet Comparisons

Liwen Zhang, Subhransu Maji, Ryota Tomioka

Abstract

Similarity between objects is multi-faceted and it can be easier for human annotators to measure it when the focus is on a specific aspect. We consider the problem of mapping objects into view-specific embeddings where the distance between them is consistent with the similarity comparisons of the form "from the t-th view, object A is more similar to B than to C". Our framework jointly learns view-specific embeddings exploiting correlations between views. Experiments on a number of datasets, including one of multi-view crowdsourced comparison on bird images, show the proposed method achieves lower triplet generalization error when compared to both learning embeddings independently for each view and all views pooled into one view. Our method can also be used to learn multiple measures of similarity over input features taking class labels into account and compares favorably to existing approaches for multi-task metric learning on the ISOLET dataset.

Introduction: The Multi-Faceted Nature of Similarity

Similarity is rarely a single scalar value. When comparing complex objects like birds, the perceived similarity depends heavily on the "view" or attribute being considered—such as size, plumage color, or beak shape. In machine learning, similarity is often learned from triplet comparisons: "Object A is more similar to B than to C." However, general similarity tasks can be ambiguous for human annotators. It is often more reliable to ask for view-specific comparisons, such as "In terms of the head, is bird A more like B or C?"

Multi-view similarity comparisons for birds.

Learning these specific views independently requires a vast amount of data. Conversely, "pooling" all triplets into a single model ignores the distinct nuances of each view. This paper proposes a framework to jointly learn these views by exploiting their underlying correlations.

Methodology: Joint Embedding via Shared Subspaces

The authors propose a joint learning model based on matrix factorization. Rather than learning an entirely independent metric for every view, they decompose the similarity measure into two components:

  1. A Global Transformation ($L$): A shared matrix that maps all objects into a common $D$-dimensional subspace.
  2. Local View-Specific Metrics ($M_t$): Small, positive semidefinite matrices that adjust the shared space for each specific view $t$.

The metric for a view is defined as $K_t = L M_t L^\top$. This structure forces different views to share a common low-rank subspace, acting as a bottleneck that prevents overfitting and facilitates information transfer between views. Optimization is performed by alternating between updating the global $L$ and the local $M_t$ using gradient descent, with trace regularization to encourage low-rank solutions.

Experimental Evaluation

The framework was evaluated across several diverse benchmarks, including the CUB-200 Birds dataset, ISOLET (spoken alphabet recognition), and PubFig (faces). For the bird dataset, crowdsourced triplets were collected where users compared specific parts like the beak, breast, or wing.

Defining views through localized image patches.

Key Results and Insights

The joint approach consistently outperformed both independent learning and pooling. On the CUB-200 bird dataset, the model achieved significantly lower triplet generalization error.

A key finding is the model's capacity for transfer learning. By utilizing the shared global space $L$, the model can learn a new view much faster than independent methods, requiring far fewer triplets to reach convergence. This "Goldilocks" approach works best when views are related but distinct—the typical regime for real-world data.

On the ISOLET dataset, the proposed method outperformed the state-of-the-art Multi-Task Large Margin Nearest Neighbor (MT-LMNN) algorithm, reducing the average test error from 5.91% to 4.78% in view-specific tasks.

Conclusion

By jointly learning multiple similarity measures through a shared low-rank subspace, this framework provides a data-efficient way to capture the multi-faceted nature of similarity. Future work may involve automatically discovering these latent views rather than relying on predefined categories, and scaling the approach to handle millions of objects.

Summary updated

Abstract

Similarity between objects is multi-faceted and it can be easier for human annotators to measure it when the focus is on a specific aspect. We consider the problem of mapping objects into view-specific embeddings where the distance between them is consistent with the similarity comparisons of the form "from the t-th view, object A is more similar to B than to C". Our framework jointly learns view-specific embeddings exploiting correlations between views. Experiments on a number of datasets, including one of multi-view crowdsourced comparison on bird images, show the proposed method achieves lower triplet generalization error when compared to both learning embeddings independently for each view and all views pooled into one view. Our method can also be used to learn multiple measures of similarity over input features taking class labels into account and compares favorably to existing approaches for multi-task metric learning on the ISOLET dataset.
BibTeX
@misc{zhang2015jointly,
  author = {Liwen Zhang and Subhransu Maji and Ryota Tomioka},
  title = {{Jointly Learning Multiple Measures of Similarities from Triplet Comparisons}},
  year = {2015},
  eprint = {1503.01521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1503.01521},
}
Figure