TL;DR
TLLC (Transfer Learning-based Label Completion) tackles the "sparse data" problem in crowdsourcing by using Siamese networks and transfer learning. By pre-training on high-confidence labels and fine-tuning for individual workers, it accurately predicts missing labels in sparse datasets, significantly boosting the performance of downstream truth-inference algorithms.
The Challenge of Sparse Crowdsourced Data
Crowdsourcing is a double-edged sword: it is a fast, cost-effective way to label massive datasets, but the results are often noisy and incomplete. To find the "ground truth," researchers use label aggregation—algorithms that weigh different workers' inputs to decide on a final label.
However, in the real world, most workers only annotate a handful of items to save time and effort. This results in a highly sparse label matrix, where most entries are missing. Recent attempts to fix this use "worker modeling" (learning a worker's specific biases or expertise) to fill in the blanks. The catch? You cannot build a reliable model of a worker if you only have two or three examples of their work. This "insufficient worker modeling" is the primary bottleneck that TLLC aims to break.
Methods: A Transfer Learning Pipeline
The authors propose a multi-stage pipeline that leverages Transfer Learning to bridge the data gap and accurately complete missing labels.
- Source and Target Domain Construction: Instead of relying on ground truth (which is unavailable), the authors use "confident learning" to identify high-confidence instances across the entire dataset. These reliable labels form the Source Domain. The sparse, individual labels from a specific worker form the Target Domain.
- Worker Modeling via Siamese Networks: The authors pre-train a Siamese network on the Source Domain. Siamese networks are designed to learn similarity; they map data into an embedding space where items with the same label are clustered close together.
- Parameter-based Transfer: The pre-trained network is then fine-tuned for each worker separately using their specific (albeit few) labels. This allows the model to capture the unique "signature" or characteristics of that worker's labeling style without needing a massive amount of data from them specifically.
- Label Completion: Finally, the system uses these worker-specific embeddings to calculate class centroids. If an unannotated instance is close enough to a class centroid in the embedding space, the missing label is filled in.
Datasets
The researchers evaluated TLLC on three real-world datasets collected via Amazon Mechanical Turk (AMT):
- Income: A binary classification task (600 instances, 85% missing labels).
- Leaves: A multi-class task with 6 categories (384 instances, 88% missing labels).
- Music_genre: A complex multi-class task with 10 categories (700 instances, 90% missing labels).
These datasets represent a range of difficulty and sparsity, providing a robust testbed for label completion.
Key Insights
- Knowledge Sharing is Key: The core insight is that while an individual worker provides very little data, the collective high-confidence labels across the whole pool contain enough "essential knowledge" to build a foundational model. This model can then be customized for individuals.
- Individuality Matters: Unlike some methods that "smooth" worker behavior (effectively assuming everyone is the same), TLLC’s fine-tuning step preserves the unique characteristics of each worker. This is crucial because a worker might be an expert in one specific sub-category but poor in others.
- Robustness to Noise: The authors provide a theoretical proof showing that their worker modeling approach is robust to independent and identically distributed (i.i.d.) Gaussian noise in the supervisory labels.
Key Results
TLLC consistently outperformed the state-of-the-art baseline, WSLC (Worker Similarity-based Label Completion), across various aggregation methods.
- Accuracy Boost: On the Income dataset, using TLLC for label completion before Majority Voting (MV) improved accuracy from ~71% to nearly 75%.
- Significance: Critical difference diagrams showed that TLLC achieved superior average rankings across multiple aggregation methods (like MV, GTIC, and LAGNN) on the Income and Leaves datasets.
- Sparsity Resilience: TLLC proved particularly effective in scenarios with very high missing rates (above 0.5), which is exactly where traditional worker modeling usually fails.
Limitations and Future Work
The primary limitation identified is a lack of robustness against "high-volume adversarial workers." These are workers who provide a large number of labels but with exceptionally low quality. Because TLLC seeks to capture a worker's unique characteristics, it inadvertently "learns" and propagates the errors of these prolific but inaccurate workers.
Future research will likely focus on refining the transfer learning process to better filter out these adversarial influences while maintaining the benefits of individualized worker modeling.
Summary updated