Bundle Optimization for Multi-aspect Embedding

Qiong Zeng, Wenzheng Chen, Zhuo Han, Mingyi Shi, Yanir Kleiman, Daniel Cohen-Or, Baoquan Chen, Yangyan LiarXiv 2017 · arXiv

Introduction: The Challenge of Subjective Similarity

Semantic similarity is inherently subjective. For instance, two images might be perceived as "similar" due to shared color palettes, identical content, or a common artistic style. Traditional metric learning often attempts to collapse these diverse perceptual signals into a single Euclidean space. This "one-size-fits-all" approach frequently results in a "muddy" average that fails to represent any single aspect accurately, as conflicting signals (e.g., "similar color" vs. "different category") cancel each other out.

The fundamental difficulty lies in the latent nature of human judgment. When a user indicates that "Image A is more like B than C," the specific criterion—the "aspect"—driving that decision remains hidden. Aggregating thousands of such triplets introduces a massive number of unknown variables: both the image coordinates in the embedding space and the specific aspect used for every individual query. This high dimensionality makes optimization highly unstable, prone to local minima, and extremely sensitive to noise.

The Bundle Optimization Framework

To address these challenges, the authors introduce Bundle Optimization, a framework designed to disentangle multiple similarity aspects by grouping human perceptual queries into "bundles" that share a single latent aspect.

Multi-aspect similarity illustrated through car rendering styles and types.

The framework relies on two primary innovations:

1. Bundled Queries via Clustering

Instead of collecting isolated triplets, the system utilizes clustering tasks. In these tasks, a user is asked to sort a group of images into several bins. Because humans naturally adopt a single consistent criterion (e.g., "shape," "function," or "color") to perform a single sorting task, all similarity relations derived from that specific task are "bundled" under the same latent aspect. This effectively "locks" the aspect variable for hundreds of triplets simultaneously.

2. Joint Optimization with Aspect Inference

The model defines a loss function that evaluates image distances across $K$ distinct embedding spaces. It introduces aspect inference variables ($\alpha$) representing the probability that a specific bundle belongs to a particular embedding space.

The algorithm uses a contrastive loss function and the Adam optimizer to simultaneously learn the coordinates of images in $K$ different spaces and assign each bundle of human data to the space it fits best. By bundling, the number of "aspect variables" is reduced from one per triplet to one per bundle, which significantly stabilizes the underlying mathematics and allows for clearer disentanglement.

Experimental Evaluation and Results

The researchers evaluated Bundle Optimization using a mix of synthetic data, rendered objects, and real-world media datasets.

Synthetic Recovery (AOB Dataset)

On the synthetic "AOB" dataset—where 214 points are distributed to form 'A', 'O', and 'B' shapes across different ground-truth embeddings—Bundle Optimization demonstrated superior recovery accuracy.

Comparison of ground truth recovery for 'A', 'O', and 'B' shapes using Bundle Optimization versus non-bundled methods.

As shown in the visualization, the bundled approach successfully reconstructed the original intended structures from randomized initializations. In contrast, non-bundled methods struggled to separate the overlapping signals, resulting in distorted shapes. Quantitatively, Bundle Optimization achieved significantly higher NDCG (Normalized Discounted Cumulative Gain) scores, indicating that the recovered rankings closely matched the ground truth.

Real-World and Benchmark Performance

  • ShapeNet Chairs: Using 6,777 rendered images, the model successfully separated embeddings into spaces that clearly represented "leg style," "arm style," and "back style" without any manual labeling of those categories.
  • Movie Posters: In a real-world application, users clustered posters based on their own preferences. The model revealed latent aspects such as color/composition, genre, and animation vs. live-action, demonstrating its ability to discover meaningful structures in unlabelled data.
  • Robustness to Noise: The method proved highly robust to human error. Because the optimization relies on the consensus of a bundle, a few "wrong" clicks by a user are outweighed by the rest of the bundle's data, making it more resilient than triplet-based methods like Multi-View Triplet Embedding (MVTE).

Discussion and Key Insights

A key "aha moment" of this research is that the natural limitation of human focus—the fact that we usually think about one attribute at a time when sorting—is actually a powerful mathematical constraint. By forcing users to cluster, the researchers transform a weakness into a feature that enables disentanglement without explicit labels. The model does not need to be told "this space is for color"; it naturally pushes color-based relations into one space to minimize total conflict.

Limitations and Future Work

While effective, the method currently requires the user to manually specify the number of embedding spaces ($K$). If $K$ is too small, aspects remain fused; if too large, the model may create redundant spaces. A promising future direction involves developing an automated method to determine the optimal number of aspects based on the "conflict level" of the collected human data. Additionally, while robust, the system's sensitivity to extreme outliers remains an area for further refinement.

Summary updated

Abstract

Understanding semantic similarity among images is the core of a wide range of computer vision applications. An important step towards this goal is to collect and learn human perceptions. Interestingly, the semantic context of images is often ambiguous as images can be perceived with emphasis on different aspects, which may be contradictory to each other. In this paper, we present a method for learning the semantic similarity among images, inferring their latent aspects and embedding them into multi-spaces corresponding to their semantic aspects. We consider the multi-embedding problem as an optimization function that evaluates the embedded distances with respect to the qualitative clustering queries. The key idea of our approach is to collect and embed qualitative measures that share the same aspects in bundles. To ensure similarity aspect sharing among multiple measures, image classification queries are presented to, and solved by users. The collected image clusters are then converted into bundles of tuples, which are fed into our bundle optimization algorithm that jointly infers the aspect similarity and multi-aspect embedding. Extensive experimental results show that our approach significantly outperforms state-of-the-art multi-embedding approaches on various datasets, and scales well for large multi-aspect similarity measures.
BibTeX
@misc{zeng2017bundle,
  author = {Qiong Zeng and Wenzheng Chen and Zhuo Han and Mingyi Shi and Yanir Kleiman and Daniel Cohen-Or and Baoquan Chen and Yangyan Li},
  title = {{Bundle Optimization for Multi-aspect Embedding}},
  year = {2017},
  eprint = {1703.09928},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1703.09928},
}
Figure