TLLC: Transfer Learning-based Label Completion for Crowdsourcing

Wenjun Zhang, Liangxiao Jiang, Chaoqun LiICML 2025 · OpenReview

TL;DR

TLLC (Transfer Learning-based Label Completion) tackles the "sparse data" problem in crowdsourcing by using Siamese networks and transfer learning. By pre-training on high-confidence labels and fine-tuning for individual workers, it accurately predicts missing labels in sparse datasets, significantly boosting the performance of downstream truth-inference algorithms.

The Challenge of Sparse Crowdsourced Data

Crowdsourcing is a double-edged sword: it is a fast, cost-effective way to label massive datasets, but the results are often noisy and incomplete. To find the "ground truth," researchers use label aggregation—algorithms that weigh different workers' inputs to decide on a final label.

However, in the real world, most workers only annotate a handful of items to save time and effort. This results in a highly sparse label matrix, where most entries are missing. Recent attempts to fix this use "worker modeling" (learning a worker's specific biases or expertise) to fill in the blanks. The catch? You cannot build a reliable model of a worker if you only have two or three examples of their work. This "insufficient worker modeling" is the primary bottleneck that TLLC aims to break.

Methods: A Transfer Learning Pipeline

The authors propose a multi-stage pipeline that leverages Transfer Learning to bridge the data gap and accurately complete missing labels.

The TLLC pipeline for label completion.
  1. Source and Target Domain Construction: Instead of relying on ground truth (which is unavailable), the authors use "confident learning" to identify high-confidence instances across the entire dataset. These reliable labels form the Source Domain. The sparse, individual labels from a specific worker form the Target Domain.
  2. Worker Modeling via Siamese Networks: The authors pre-train a Siamese network on the Source Domain. Siamese networks are designed to learn similarity; they map data into an embedding space where items with the same label are clustered close together.
  3. Parameter-based Transfer: The pre-trained network is then fine-tuned for each worker separately using their specific (albeit few) labels. This allows the model to capture the unique "signature" or characteristics of that worker's labeling style without needing a massive amount of data from them specifically.
  4. Label Completion: Finally, the system uses these worker-specific embeddings to calculate class centroids. If an unannotated instance is close enough to a class centroid in the embedding space, the missing label is filled in.

Datasets

The researchers evaluated TLLC on three real-world datasets collected via Amazon Mechanical Turk (AMT):

  • Income: A binary classification task (600 instances, 85% missing labels).
  • Leaves: A multi-class task with 6 categories (384 instances, 88% missing labels).
  • Music_genre: A complex multi-class task with 10 categories (700 instances, 90% missing labels).

These datasets represent a range of difficulty and sparsity, providing a robust testbed for label completion.

Key Insights

  • Knowledge Sharing is Key: The core insight is that while an individual worker provides very little data, the collective high-confidence labels across the whole pool contain enough "essential knowledge" to build a foundational model. This model can then be customized for individuals.
  • Individuality Matters: Unlike some methods that "smooth" worker behavior (effectively assuming everyone is the same), TLLC’s fine-tuning step preserves the unique characteristics of each worker. This is crucial because a worker might be an expert in one specific sub-category but poor in others.
  • Robustness to Noise: The authors provide a theoretical proof showing that their worker modeling approach is robust to independent and identically distributed (i.i.d.) Gaussian noise in the supervisory labels.

Key Results

TLLC consistently outperformed the state-of-the-art baseline, WSLC (Worker Similarity-based Label Completion), across various aggregation methods.

Performance comparison of TLLC vs. WSLC across different datasets and aggregation methods.
  • Accuracy Boost: On the Income dataset, using TLLC for label completion before Majority Voting (MV) improved accuracy from ~71% to nearly 75%.
  • Significance: Critical difference diagrams showed that TLLC achieved superior average rankings across multiple aggregation methods (like MV, GTIC, and LAGNN) on the Income and Leaves datasets.
  • Sparsity Resilience: TLLC proved particularly effective in scenarios with very high missing rates (above 0.5), which is exactly where traditional worker modeling usually fails.

Limitations and Future Work

The primary limitation identified is a lack of robustness against "high-volume adversarial workers." These are workers who provide a large number of labels but with exceptionally low quality. Because TLLC seeks to capture a worker's unique characteristics, it inadvertently "learns" and propagates the errors of these prolific but inaccurate workers.

Future research will likely focus on refining the transfer learning process to better filter out these adversarial influences while maintaining the benefits of individualized worker modeling.

Summary updated

Abstract

Label completion serves as a preprocessing approach to handling the sparse crowdsourced label matrix problem, significantly boosting the effectiveness of the downstream label aggregation. In recent advances, worker modeling has been proved to be a powerful strategy to further improve the performance of label completion. However, in real-world scenarios, workers typically annotate only a few instances, leading to insufficient worker modeling and thus limiting the improvement of label completion. To address this issue, we propose a novel transfer learning-based label completion (TLLC) method. Specifically, we first identify all high-confidence instances from the whole crowdsourced data as a source domain and use it to pretrain a Siamese network. The abundant annotated instances in the source domain provide essential knowledge for worker modeling. Then, we transfer the pretrained network to the target domain with the instances annotated by each worker separately, ensuring worker modeling captures unique characteristics of each worker. Finally, we leverage the new embeddings learned by the transferred network to complete each worker’s missing labels. Extensive experiments on several widely used real-world datasets demonstrate the effectiveness of TLLC. Our codes and datasets are available at https://github.com/jiangliangxiao/TLLC.
BibTeX
@inproceedings{zhang2025tllc,
  author = {Wenjun Zhang and Liangxiao Jiang and Chaoqun Li},
  title = {{TLLC: Transfer Learning-based Label Completion for Crowdsourcing}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=BkdAnSKNoX},
}
Figure