Soft Target-Enhanced Matching Framework for Deep Entity Matching

Wenzhou Dou, Derong Shen, Xiangmin Zhou, Tiezheng Nie, Yue Kou, Hang Cui, Ge YuAAAI 2023 · OpenReview

TL;DR

This paper introduces STEAM, a novel framework for deep entity matching that improves model generalization by using "soft targets" generated by a previous model iteration as a regularizer. This approach effectively balances bias and variance, leading to better performance on unseen data without requiring model redesign or extra computational cost.

What Problem Does This Paper Address?

Entity Matching (EM) is crucial for data integration, aiming to identify records that refer to the same real-world entity. Existing deep learning-based EM models often rely on "one-hot" labels (hard, definitive labels) during training. While this provides a clear signal, it can lead to models that overfit the training data and perform poorly on new, unseen data. This is because the training data might not fully represent the diversity of real-world data, leading to a poor bias-variance tradeoff. The paper addresses this by proposing a method to create more generalized EM models that perform well even on data distributions different from the training set.

Methods

The core of the proposed solution is the Soft Target-Enhanced Matching (STEAM) framework. STEAM operates within a standard deep entity matching pipeline but introduces a novel model training pipeline focused on "soft supervised training."

Here's how it works:

  • Virtual Teacher: In each training iteration, the current EM model acts as a "virtual teacher."
  • Soft Targets: The virtual teacher's predictions are "softened" using a temperature parameter in a softmax function. These softened outputs serve as "soft labels."
  • Label-wise Regularization: These soft labels are then used as a regularizer to train the EM model in the current iteration. This is inspired by knowledge distillation but implemented in a teacher-free manner.
  • Bias-Variance Tradeoff: The soft labels provide a more nuanced supervision signal than hard one-hot labels, helping to balance the model's tendency to fit the training data too closely (low bias, high variance) versus its ability to generalize (high bias, low variance).
  • Sample-wise Soft Weighting (SSW): To further refine this, the paper introduces an adaptive weighting mechanism. It quantifies the bias and variance for each training sample and dynamically adjusts the weight of the soft training loss. Samples that are "hard-to-fit" (where the model's prediction is far from the ground truth) receive a lower weight for the soft loss, preventing them from overly influencing the regularization.

Datasets

The paper evaluates STEAM on two well-established entity matching benchmarks:

  • WDC Benchmark: This benchmark consists of four subsets: Computers, Cameras, Watches, and Shoes. The experiments use different scaling factors (Small, Medium, Large, xLarge) to assess label efficiency.
  • DeepMatcher Benchmark: This benchmark includes challenging and representative datasets: Amazon-Google (A-G), Walmart-Amazon (W-A) and its dirty version (W-A*), and Abt-Buy (A-B). These cover structured, dirty, and textual data scenarios.

Key Insights and Ideas

  • The Problem with Hard Labels: The paper strongly argues that sharp, one-hot supervision signals in deep EM models are a primary cause of poor generalization. They lead to overfitting and a detrimental bias-variance tradeoff.
  • Soft Targets as Regularizers: The core innovation is using softened predictions from a previous model iteration as a form of label-wise regularization. This is a clever way to leverage existing model capabilities to improve future iterations without needing new data or complex model architectures.
  • Teacher-Free Knowledge Distillation: The implementation of soft targets without an explicit, separate "teacher" model is an elegant simplification, making the approach more practical.
  • Adaptive Weighting for Hard-to-Fit Samples: The Sample-wise Soft Weighting (SSW) is a crucial refinement. It acknowledges that not all samples benefit equally from soft regularization and intelligently down-weights problematic samples to prevent negative interference.
  • Generalization is Key: The overarching theme is that achieving good performance on unseen data is paramount, and STEAM directly targets this by improving the model's ability to generalize.

Key Results

  • Superior Performance: STEAM consistently outperforms state-of-the-art baselines (including strong PLM-based models like Ditto and HierGAT) across all subsets of the WDC and DeepMatcher benchmarks.
  • Label Efficiency: STEAM demonstrates significant label efficiency. For instance, on the WDC-Small dataset (using only 1/20 of the data), STEAM achieves competitive results compared to Ditto using 1/8 of the data.
  • Improved Effectiveness: The framework leads to higher F1 scores, indicating more accurate identification of matching record pairs. For example, the average improvement (ΔF1ave) against Ditto on WDC datasets ranges from 1.30 to 7.93.
  • Effective Bias-Variance Control: Visualizations show that STEAM-SSW effectively controls both bias and variance, leading to tighter clusters and more distinct decision boundaries compared to a baseline without soft regularization.

Limitations and Future Work

  • Computational Cost: While the paper states STEAM achieves its goals "without any additional computational cost," the iterative nature of using a previous model's output as a teacher implies a dependency on prior training steps. The paper focuses on the model architecture not having additional cost, but the training process itself is iterative.
  • Hyperparameter Tuning: The paper mentions setting the temperature $\tau=10$ and $\lambda=1$ for experiments. Further exploration into the sensitivity and optimal tuning of these hyperparameters across diverse datasets could be beneficial.
  • Future Work: The authors suggest that their framework can be extended to other deep learning tasks where generalization is a challenge. The core idea of soft target regularization is broadly applicable.

Summary updated

Abstract

Deep Entity Matching (EM) is one of the core research topics in data integration. Typical existing works construct EM models by training deep neural networks (DNNs) based on the training samples with onehot labels. However, these sharp supervision signals of onehot labels harm the generalization of EM models, causing them to overfit the training samples and perform badly in unseen datasets. To solve this problem, we first propose that the challenge of training a well-generalized EM model lies in achieving the compromise between fitting the training samples and imposing regularization, i.e., the bias-variance tradeoff. Then, we propose a novel Soft Target-EnhAnced Matching (Steam) framework, which exploits the automatically generated soft targets as label-wise regularizers to constrain the model training. Specifically, Steam regards the EM model trained in previous iteration as a virtual teacher and takes its softened output as the extra regularizer to train the EM model in the current iteration. As such, Steam effectively calibrates the obtained EM model, achieving the bias-variance tradeoff without any additional computational cost. We conduct extensive experiments over open datasets and the results show that our proposed Steam outperforms the state-of-the-art EM approaches in terms of effectiveness and label efficiency.
BibTeX
@inproceedings{dou2023soft,
  author = {Wenzhou Dou and Derong Shen and Xiangmin Zhou and Tiezheng Nie and Yue Kou and Hang Cui and Ge Yu},
  title = {{Soft Target-Enhanced Matching Framework for Deep Entity Matching}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=FHESA2VyVk},
}
Figure