Abstract
Similarity between objects is multi-faceted and it can be easier for human annotators to measure it when the focus is on a specific aspect. We consider the problem of mapping objects into view-specific embeddings where the distance between them is consistent with the similarity comparisons of the form "from the t-th view, object A is more similar to B than to C". Our framework jointly learns view-specific embeddings exploiting correlations between views. Experiments on a number of datasets, including one of multi-view crowdsourced comparison on bird images, show the proposed method achieves lower triplet generalization error when compared to both learning embeddings independently for each view and all views pooled into one view. Our method can also be used to learn multiple measures of similarity over input features taking class labels into account and compares favorably to existing approaches for multi-task metric learning on the ISOLET dataset.
Introduction: The Multi-Faceted Nature of Similarity
When we say two things are "similar," we are usually focusing on a specific aspect. For example, two birds might have similar beaks but completely different wing patterns. Standard machine learning approaches often try to learn a single, "average" similarity metric. However, this leads to noisy data because human annotators might be looking at different features when making judgments.
While one could learn separate embeddings for every possible "view" (e.g., one for beaks, one for wings), this is incredibly data-hungry. Learning an embedding for $N$ objects can require $O(N^3)$ triplet comparisons in the worst case. This paper tackles the challenge of learning these multiple facets of similarity simultaneously, exploiting the fact that different views of the same object are often highly correlated.
Methods: Joint Metric Learning
The proposed framework combines explicit global embeddings with implicit view-specific metrics to share information across perspectives. The model consists of:
- Global Embedding ($x_i$): Each object is mapped to a vector in a shared $D$-dimensional space, representing its "core" identity.
- View-Specific Metrics ($M_t$): For each specific perspective (or "view") $t$, the model learns a Mahalanobis distance matrix. This matrix acts like a "lens" that projects the global embedding into a space relevant to that specific view.
- Joint Optimization: The model is trained using "triplet comparisons" (e.g., "From the perspective of the head, bird A is more similar to B than to C"). The objective function minimizes a loss (like hinge loss) that penalizes violations of these triplets across all views simultaneously.
By assuming each view is a low-rank projection of a common global embedding, the model drastically reduces the number of parameters to fit, moving from $O(NDT)$ for independent views to $O(ND + D^2T)$.
Experimental Evaluation
The researchers evaluated their method on four distinct types of data:
- Synthetic Data: Controlled datasets with both uniform and clustered distributions.
- Poses of Airplanes: 200 images from the PASCAL VOC dataset. Views were defined by subsets of 16 landmarks (e.g., "nose," "back + wings").
- PubFig (Public Figures Face Database): 200 images categorized by 39 attributes grouped into five views: hair, age, accessory, shape, and ethnicity.
- CUB-200 Birds: 200 species of birds. This dataset featured crowd-sourced triplets where humans compared birds based on parts like the beak, breast, or wing.
Key Results and Insights
- Superior Generalization: In almost all cases, the joint learning approach achieved lower triplet generalization error than learning views independently.
- Data Efficiency: On synthetic data, joint learning reached a target error rate with roughly 1/4th the number of triplets required by independent learning. The "Shared Global View" acts as a regularizer; knowledge learned about a bird's identity from "wing" triplets helps refine the "beak" similarity even if beak-specific data is sparse.
- Better Clustering: The learned embeddings showed better "category-level" structure. Even without class labels during training, the resulting embeddings grouped species together more effectively than baseline methods.
- "Zero-Shot" Transfer: In a simulated "new view" scenario on the bird dataset, the model could produce a meaningful embedding for a new part using only 100 triplets, provided it had seen enough triplets from other parts.
Conclusion
By explicitly asking for view-specific feedback, the researchers reduce the noise inherent in "general" similarity judgments. While joint learning introduces a bias (the assumption that views are related), this bias significantly reduces error when training data is limited. Future work may explore how these multi-view embeddings can be used for interactive classification tasks, where a system asks a user for specific comparisons to quickly narrow down an object's identity.
Summary updated