Retrieval-Enhanced Dual Encoder Training for Product Matching

Justin ChiuEMNLP 2023

TL;DR

This paper introduces a novel two-stage training method for dual-encoder models used in product matching. By intelligently filtering and selecting training data in the second stage, they achieve significant improvements in matching accuracy on both public and real-world e-commerce datasets, demonstrating the importance of curated training data beyond just more data.

What Problem Does This Paper Address?

Product matching, the task of identifying the same product across different listings (e.g., a specific shirt sold by multiple vendors), is crucial for e-commerce platforms. This task needs to be highly efficient for large-scale operations. Dual-encoder models have emerged as a powerful solution, encoding product information into dense vectors for fast similarity comparisons. However, training these models effectively requires carefully curated positive (matching) and negative (non-matching) pairs. Traditional methods like "in-batch negative" can generate many "trivial" negative pairs that don't help the model learn, and simply adding more positive pairs doesn't always guarantee better performance. The challenge is to create a more effective training dataset that leads to a more robust dual-encoder model.

Methods

The paper proposes a "Retrieval-Enhanced Dual Encoder Training" approach, which consists of two stages:

  • Stage 1: Data Pre-filtering and Pair Generation: A preliminary dual-encoder model is trained using existing human-annotated positive pairs and in-batch negatives. This model is then used to retrieve potential matches for the queries from the training set across a larger collection of targets. The top N retrieved results are used to form the training data for Stage 2. This process aims to identify more informative positive pairs and potentially exclude less useful ones from Stage 1.
  • Stage 2: Training with Pre-filtered Data: A new dual-encoder model is trained using the curated dataset generated in Stage 1. This stage leverages the refined set of positive and negative pairs, which are expected to be more informative and less trivial than those generated by standard in-batch negative sampling.

The core novelty lies in the intelligent selection of training data for Stage 2, going beyond simply using all available data or standard hard-negative mining.

Datasets

The authors evaluate their approach on two datasets:

  • WDC Product Data Corpus: A publicly available, large-scale product matching benchmark dataset.
  • In-house English Catalog: A real-world e-commerce dataset containing 47 million products, used to demonstrate the approach's effectiveness in a production-like environment.

Key Insights and Ideas

  • Curated data beats raw data: The experiments show that removing seemingly "positive" pairs from Stage 1 training data can actually improve the performance of the Stage 2 model. This suggests that not all positive pairs are equally beneficial, and some might even be detrimental if they are too dissimilar or noisy.
  • Automated data selection is key: The retrieval-enhanced approach provides an automated way to identify and exclude less useful training data, reducing reliance on manual annotation or heuristic filtering.
  • Stage 1 as a data curator: The initial dual-encoder in Stage 1 acts not just as a preliminary model but as a sophisticated data selection tool, identifying more relevant pairs for the final training.

Key Results

The proposed retrieval-enhanced dual-encoder training method achieved significant improvements:

  • WDC Corpus: A 2% F1 score improvement on the public dataset.
  • In-house English Catalog: A substantial 9% F1 score improvement on the real-world e-commerce dataset.
  • The Stage 2 dual-encoder, trained with the curated data, offers a better balance of precision and recall compared to the Stage 1 model, while still being computationally more efficient during inference than heavy BERT-based joint encoding models.

Limitations and Future Work

  • Threshold Sensitivity: The paper notes that performance can be sensitive to the threshold used for selecting retrieved results in Stage 1. The optimal threshold might vary across datasets.
  • Analysis of Excluded Pairs: While the paper shows that excluding Stage 1 positive pairs improves performance, a deeper analysis into why these specific pairs are detrimental could be beneficial. The authors suggest that adding back excluded pairs can degrade performance, indicating the model's ability to identify and discard noisy data.
  • Further Exploration of Data Selection: The authors imply that this method of identifying and excluding less useful training data is a promising direction for future research in improving model training.

Summary updated

BibTeX
@inproceedings{chiu2023retrieval,
  author = {Justin Chiu},
  title = {{Retrieval-Enhanced Dual Encoder Training for Product Matching}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure