Multiview Representation Learning from Crowdsourced Triplet Comparisons

Xiaotian Lu, Jiyi Li, Koh Takeuchi, Hisashi Kashima

TL;DR

Researchers have developed a new deep learning framework that learns multi-attribute object representations (like color and shape) using only simple "which two are more similar?" triplet comparisons from crowd workers. Unlike previous methods, this approach is inductive, meaning it can generate these "multiview" embeddings for brand-new images it has never seen before, while accounting for the fact that different people focus on different features.

What Problem Does This Paper Address?

When we ask crowd workers to label data, we often ask for categories (e.g., "Is this a Golden Retriever?"). However, categorization is hard and requires expertise. A simpler alternative is triplet comparison: "Among images A, B, and C, which two are most similar?"

The catch? Similarity is subjective. If shown a red circle, a red square, and a blue circle, one worker might pair the first two based on color, while another might pair the first and third based on shape. These different attributes are called "views."

Existing algorithms for learning from these triplets have two major flaws:

  1. Transductive Limitation: They can only calculate embeddings for the specific images used during training. They can't handle a new image without retraining the whole system.
  2. Worker Bias: They often struggle to reconcile the fact that different workers have different preferences for which "view" (attribute) matters most.

Methods

The authors propose an end-to-end deep learning framework that moves from a shared "backbone" to multiple parallel "view" branches.

  • The Architecture: They use a ResNet18 backbone as a shared feature extractor. The network then splits into $V$ different branches, where each branch is responsible for learning a specific "view" or attribute of the data.
  • View Selection Mechanism: This is the "secret sauce." To decide which view to prioritize for a specific triplet, the model combines two factors:
  • Inherent Weight (Triplet Entropy): It calculates how "obvious" a similarity is within a specific view. If a view makes a choice very clear (low entropy), it gets more weight.
  • Worker Preference: The model learns a unique profile for every crowd worker, identifying if they generally prefer focusing on color, shape, or other latent features.
  • Loss Function: The system is trained by maximizing the log-likelihood that the model’s weighted combination of views matches the actual choices made by the crowd workers.

Datasets

The study utilized two primary datasets to test both controlled simulations and real-world noise:

  1. 10-color MNIST: A modified version of the classic handwritten digits dataset. Each digit is assigned one of ten colors. This creates two very clear, competing views: digit identity and color.
  2. Stanford Dogs: A subset of 1,000 images across 20 breeds. This represents a more complex real-world scenario where "views" (like ear shape, fur texture, or size) are latent and not explicitly defined.

The researchers collected data from 160 real crowd workers via the Lancers platform to validate the model against human subjectivity.

Key Insights and Ideas

  • Ambiguity as Information: Instead of trying to "average out" worker disagreements, the model treats disagreement as evidence of multiple valid attributes.
  • Semantic Separation: One of the most striking findings was that the different branches of the network naturally "specialized." In the MNIST tests, without being told to do so, one branch learned to represent colors while the other learned to represent digit shapes.
  • Inductive Power: Because this is a neural network, once trained, you can drop in a completely new image, and the model will instantly output a vector for its color and a separate vector for its shape.
  • The Role of Entropy: The ablation study proved that "triplet entropy" is vital. By measuring how "confused" a view is by a certain triplet, the model can effectively ignore irrelevant attributes for specific tasks.

Key Results

The multiview approach significantly outperformed single-view baselines across all metrics:

  • Accuracy: On the real-world 10-color MNIST crowd data, the multiview model achieved 69.4% accuracy, compared to just 45.8% for the single-view baseline.
  • Clustering: Using K-means on the learned embeddings, the multiview model showed much higher "Purity" and "Normalized Mutual Information" (NMI), proving the embeddings were more semantically meaningful.
  • Worker Modeling: The model successfully "learned" the preferences of simulated workers, correctly identifying which workers were "color-focused" versus "number-focused."

Limitations and Future Work

  • The "V" Problem: Currently, the number of views ($V$) is a hyperparameter that must be set in advance. The model cannot yet automatically determine how many distinct attributes exist in a dataset.
  • Backbone Dependency: The results are tied to the ResNet18 architecture. The authors suggest that exploring other network structures (like Transformers) could yield even better feature extraction.
  • Latent View Interpretation: While the model separates views in the "Dog" dataset, it is still difficult for humans to perfectly describe what specific attribute a latent "Dog View" represents (e.g., is it "snout length" or "ear floppiness"?). Future work could focus on making these learned views more interpretable.

Summary updated

Abstract

Crowdsourcing has been used to collect data at scale in numerous fields. Triplet similarity comparison is a type of crowdsourcing task, in which crowd workers are asked the question “among three given objects, which two are more similar?”, which is relatively easy for humans to answer. However, the comparison can be sometimes based on multiple views, i.e., different independent attributes such as color and shape. Each view may lead to different results for the same three objects. Although an algorithm was proposed in prior work to produce multiview embeddings, it involves at least two problems: (1) the existing algorithm cannot independently predict multiview embeddings for a new sample, and (2) different people may prefer different views. In this study, we propose an end-to-end inductive deep learning framework to solve the multiview representation learning problem. The results show that our proposed method can obtain multiview embeddings of any object, in which each view corresponds to an independent attribute of the object. We collected two datasets from a crowdsourcing platform to experimentally investigate the performance of our proposed approach compared to conventional baseline methods.
BibTeX
@inproceedings{lu2023multiview,
  author = {Xiaotian Lu and Jiyi Li and Koh Takeuchi and Hisashi Kashima},
  title = {{Multiview Representation Learning from Crowdsourced Triplet Comparisons}},
  booktitle = {The Web Conference},
  year = {2023},
  doi = {10.1145/3543507.3583431},
}

Loading related content...

Figure