Multiview Representation Learning from Crowdsourced Triplet Comparisons

Xiaotian Lu, Jiyi Li, Koh Takeuchi, Hisashi Kashima

Abstract

Crowdsourcing has been used to collect data at scale in numerous fields. Triplet similarity comparison is a type of crowdsourcing task, in which crowd workers are asked the question "among three given objects, which two are more similar?", which is relatively easy for humans to answer. However, the comparison can be sometimes based on multiple views, i.e., different independent attributes such as color and shape. Each view may lead to different results for the same three objects. Although an algorithm was proposed in prior work to produce multiview embeddings, it involves at least two problems: (1) the existing algorithm cannot independently predict multiview embeddings for a new sample, and (2) different people may prefer different views. In this study, we propose an end-to-end inductive deep learning framework to solve the multiview representation learning problem. The results show that our proposed method can obtain multiview embeddings of any object, in which each view corresponds to an independent attribute of the object. We collected two datasets from a crowdsourcing platform to experimentally investigate the performance of our proposed approach compared to conventional baseline methods.

Introduction

When labeling data at scale, researchers often rely on triplet comparisons: "Among images A, B, and C, which two are most similar?" While simpler for humans than direct categorization, this task is inherently subjective. A worker might pair objects based on color, while another might focus on shape. These distinct attributes are referred to as "views."

Existing methods for learning from these triplets face two major hurdles. First, they are often transductive, meaning they cannot generate embeddings for new, unseen images without retraining. Second, they struggle to account for worker bias—the tendency of individual contributors to favor specific attributes over others. This paper introduces an inductive deep learning framework designed to learn multiview representations while explicitly modeling worker preferences.

Methodology

The proposed framework utilizes an end-to-end architecture that transitions from a shared feature extractor to parallel, view-specific branches.

The proposed end-to-end inductive deep learning framework for multiview representation learning.

As shown in the architecture diagram, the model employs a ResNet18 backbone as a shared "backbone" ($g$) to extract general features. The network then splits into $V$ parallel branches ($h^{[1]}...h^{[V]}$), each dedicated to learning a specific attribute or "view." A key innovation is the View Selection Mechanism, which weights these views based on two factors:

  1. Triplet Entropy: The model calculates the clarity of a similarity choice within a specific view; low-entropy (obvious) choices are weighted more heavily.
  2. Worker Preference: The model learns a unique profile for every crowd worker, identifying their latent preference for specific attributes (e.g., color vs. shape).

The system is trained by maximizing the log-likelihood that the model’s weighted combination of views aligns with the actual choices made by crowd workers.

Experimental Setup

The framework was evaluated using two primary datasets:

  • 10-color MNIST: A modified handwritten digits dataset where each digit has a specific color, creating two competing views: digit identity and color.
  • Stanford Dogs: A dataset of 1,000 images across 20 breeds, representing a complex real-world scenario with latent views like fur texture and snout length.

Data was collected from 160 real crowd workers via the Lancers platform to validate the model against genuine human subjectivity.

Results and Discussion

The multiview approach significantly outperformed single-view baselines. On real-world 10-color MNIST data, the model achieved 69.4% accuracy, a substantial improvement over the 45.8% achieved by single-view methods. Clustering metrics (Purity and NMI) also confirmed that the learned embeddings were more semantically meaningful.

t-SNE visualizations of learned embeddings across different views and simulation settings.

Visualizations using t-SNE (above) demonstrate the model's ability to achieve semantic separation. In the MNIST experiments, the network naturally specialized: one branch learned to represent colors while the other represented digit shapes, without explicit supervision. Furthermore, the model successfully "learned" the preferences of simulated workers, correctly identifying those who were color-focused versus those who were number-focused.

Conclusion

This study demonstrates that an inductive deep learning framework can effectively disentangle multiple attributes from simple triplet comparisons. By treating worker disagreement as evidence of multiple valid views rather than noise, the model generates robust, multi-attribute representations for both known and novel objects. Future work will address the challenge of automatically determining the optimal number of views ($V$) and improving the interpretability of latent attributes in complex datasets like Stanford Dogs.

Summary updated

Abstract

Crowdsourcing has been used to collect data at scale in numerous fields. Triplet similarity comparison is a type of crowdsourcing task, in which crowd workers are asked the question ``among three given objects, which two are more similar?'', which is relatively easy for humans to answer. However, the comparison can be sometimes based on multiple views, i.e., different independent attributes such as color and shape. Each view may lead to different results for the same three objects. Although an algorithm was proposed in prior work to produce multiview embeddings, it involves at least two problems: (1) the existing algorithm cannot independently predict multiview embeddings for a new sample, and (2) different people may prefer different views. In this study, we propose an end-to-end inductive deep learning framework to solve the multiview representation learning problem. The results show that our proposed method can obtain multiview embeddings of any object, in which each view corresponds to an independent attribute of the object. We collected two datasets from a crowdsourcing platform to experimentally investigate the performance of our proposed approach compared to conventional baseline methods.
BibTeX
@inproceedings{lu2023multiview,
  author = {Xiaotian Lu and Jiyi Li and Koh Takeuchi and Hisashi Kashima},
  title = {{Multiview Representation Learning from Crowdsourced Triplet Comparisons}},
  booktitle = {The Web Conference},
  year = {2023},
  doi = {10.1145/3543507.3583431},
}
Figure