Introduction: The Challenge of Subjective Similarity
Semantic similarity is inherently subjective. For instance, two images might be perceived as "similar" due to shared color palettes, identical content, or a common artistic style. Traditional metric learning often attempts to collapse these diverse perceptual signals into a single Euclidean space. This "one-size-fits-all" approach frequently results in a "muddy" average that fails to represent any single aspect accurately, as conflicting signals (e.g., "similar color" vs. "different category") cancel each other out.
The fundamental difficulty lies in the latent nature of human judgment. When a user indicates that "Image A is more like B than C," the specific criterion—the "aspect"—driving that decision remains hidden. Aggregating thousands of such triplets introduces a massive number of unknown variables: both the image coordinates in the embedding space and the specific aspect used for every individual query. This high dimensionality makes optimization highly unstable, prone to local minima, and extremely sensitive to noise.
The Bundle Optimization Framework
To address these challenges, the authors introduce Bundle Optimization, a framework designed to disentangle multiple similarity aspects by grouping human perceptual queries into "bundles" that share a single latent aspect.
The framework relies on two primary innovations:
1. Bundled Queries via Clustering
Instead of collecting isolated triplets, the system utilizes clustering tasks. In these tasks, a user is asked to sort a group of images into several bins. Because humans naturally adopt a single consistent criterion (e.g., "shape," "function," or "color") to perform a single sorting task, all similarity relations derived from that specific task are "bundled" under the same latent aspect. This effectively "locks" the aspect variable for hundreds of triplets simultaneously.
2. Joint Optimization with Aspect Inference
The model defines a loss function that evaluates image distances across $K$ distinct embedding spaces. It introduces aspect inference variables ($\alpha$) representing the probability that a specific bundle belongs to a particular embedding space.
The algorithm uses a contrastive loss function and the Adam optimizer to simultaneously learn the coordinates of images in $K$ different spaces and assign each bundle of human data to the space it fits best. By bundling, the number of "aspect variables" is reduced from one per triplet to one per bundle, which significantly stabilizes the underlying mathematics and allows for clearer disentanglement.
Experimental Evaluation and Results
The researchers evaluated Bundle Optimization using a mix of synthetic data, rendered objects, and real-world media datasets.
Synthetic Recovery (AOB Dataset)
On the synthetic "AOB" dataset—where 214 points are distributed to form 'A', 'O', and 'B' shapes across different ground-truth embeddings—Bundle Optimization demonstrated superior recovery accuracy.
As shown in the visualization, the bundled approach successfully reconstructed the original intended structures from randomized initializations. In contrast, non-bundled methods struggled to separate the overlapping signals, resulting in distorted shapes. Quantitatively, Bundle Optimization achieved significantly higher NDCG (Normalized Discounted Cumulative Gain) scores, indicating that the recovered rankings closely matched the ground truth.
Real-World and Benchmark Performance
- ShapeNet Chairs: Using 6,777 rendered images, the model successfully separated embeddings into spaces that clearly represented "leg style," "arm style," and "back style" without any manual labeling of those categories.
- Movie Posters: In a real-world application, users clustered posters based on their own preferences. The model revealed latent aspects such as color/composition, genre, and animation vs. live-action, demonstrating its ability to discover meaningful structures in unlabelled data.
- Robustness to Noise: The method proved highly robust to human error. Because the optimization relies on the consensus of a bundle, a few "wrong" clicks by a user are outweighed by the rest of the bundle's data, making it more resilient than triplet-based methods like Multi-View Triplet Embedding (MVTE).
Discussion and Key Insights
A key "aha moment" of this research is that the natural limitation of human focus—the fact that we usually think about one attribute at a time when sorting—is actually a powerful mathematical constraint. By forcing users to cluster, the researchers transform a weakness into a feature that enables disentanglement without explicit labels. The model does not need to be told "this space is for color"; it naturally pushes color-based relations into one space to minimize total conflict.
Limitations and Future Work
While effective, the method currently requires the user to manually specify the number of embedding spaces ($K$). If $K$ is too small, aspects remain fused; if too large, the model may create redundant spaces. A promising future direction involves developing an automated method to determine the optimal number of aspects based on the "conflict level" of the collected human data. Additionally, while robust, the system's sensitivity to extreme outliers remains an area for further refinement.
Summary updated