Learning an accurate entity resolution model from crowdsourced labels

Jingjing Wang, Satoshi Oyama, Masahito Kurihara, Hisashi KashimaICUIMC 2014 · OpenReview

Introduction

Entity Resolution (ER) is a fundamental task in data integration and cleaning, focused on identifying whether different records in a dataset refer to the same real-world entity. For instance, determining if a web page about "Abby Watkins" refers to a specific mountain climber or a textile designer is crucial for search engine accuracy. While supervised machine learning models excel at this task, they typically require large volumes of high-quality labeled data. Obtaining such data from experts is often prohibitively expensive.

Crowdsourcing platforms like Amazon Mechanical Turk or Lancers offer a cost-effective alternative. However, crowdsourced labels are inherently noisy, as workers vary in expertise, attention, and interpretation of ambiguous cases. Traditional approaches often rely on Majority Voting (MV) to aggregate these labels into a single "consensus" before training. This paper argues that such early aggregation discards valuable information and proposes a novel "multi-Lap" method to learn directly from the raw, multi-worker data.

Methodology: The Multi-Lap Approach

The authors propose a method based on Locality Preserving Projections (LPP), a dimensionality reduction technique that maintains the local structure of data by keeping "linked" items close in a lower-dimensional space.

From Consensus to Multiple Laplacians

In a standard LPP setup, a single Laplacian matrix is constructed from consensus labels (e.g., via Majority Voting). The proposed multi-Lap (Multiple Laplacians) method instead creates a separate Laplacian matrix for every individual worker. The model then learns an optimal projection by combining these matrices, effectively preserving the diverse perspectives of the entire crowd.

Worker Weighting Schemes

To account for varying worker reliability, the researchers evaluated three weighting strategies for combining the individual matrices:

  1. Uniform (Uni): All workers are assigned equal weight.
  2. Similarity (Sim): Weights are proportional to how often a worker's labels agree with the majority.
  3. Mutual Dependency (Mut): A recursive model where worker credibility and pair relatedness are estimated simultaneously. High credibility is awarded to workers who consistently provide labels that align with highly probable links.

Experimental Setup

The study utilized the WEPS (Searching Information about Entities in the Web) dataset, focusing on web pages associated with 10 distinct person names. Features were extracted using tf-idf vectorization. Crowdsourced labels were collected via the Lancers platform, where workers grouped URLs referring to the same individual.

Distribution of worker performance for the 'Michael Howard' dataset.

As shown in the worker performance distributions (e.g., Figure 4 and Figure 5), the quality of crowd contributions varies significantly across different tasks and entities. Some datasets exhibit a high concentration of reliable workers, while others show a broader spread of performance, necessitating a robust model like multi-Lap.

Worker performance distributions for 'David Lodge' and 'Jerry Hobbs' datasets.

Results and Discussion

The experimental results demonstrate the superiority of the multi-Lap approach, particularly when combined with the Mutual Dependency (Mut) weighting scheme.

Performance Gains

The multi-Lap (Mut) method achieved the highest accuracy in 5 out of the 10 name datasets. A notable example is the "Thomas Baker" dataset, where the Multi-Lap (Mut) model reached an AUC of 0.691, compared to 0.533 for the baseline LPP model using Majority Voting.

Surpassing the Gold Standard

A key finding is that the multi-Lap method occasionally outperformed models trained on expert "Gold Standard" labels. The authors attribute this to the natural regularization provided by the crowd's varied opinions. While experts may be forced to provide binary labels for highly ambiguous cases—potentially leading to overfitting—the distribution of crowd labels captures the inherent uncertainty of the data.

Robustness and Efficiency

The method proved resilient even in datasets with lower average worker quality (e.g., "Abby Watkins"). Furthermore, the computational overhead was minimal; the multi-Lap algorithm typically added less than 0.1 seconds to the training time compared to the baseline, making it highly suitable for practical, large-scale applications.

Conclusion

This research highlights the benefits of preserving individual worker perspectives when learning from crowdsourced data for Entity Resolution. By avoiding premature label aggregation and employing sophisticated weighting schemes like Mutual Dependency, the multi-Lap method builds more robust models that can match or exceed the performance of those trained on expensive expert data. Future work will focus on developing more universal weighting measures and optimizing the algorithm for even larger datasets.

Summary updated

BibTeX
@inproceedings{wang2014learning,
  author = {Jingjing Wang and Satoshi Oyama and Masahito Kurihara and Hisashi Kashima},
  title = {{Learning an accurate entity resolution model from crowdsourced labels}},
  booktitle = {ICUIMC},
  year = {2014},
  url = {https://openreview.net/forum?id=zWgPk1wqhX},
}
Figure