Alleviate Dataset Shift Problem in Fine-grained Entity Typing with Virtual Adversarial Training

Haochen Shi, Siliang Tang, Xiaotao Gu, Bo Chen, Zhigang Chen, Jian Shao, Xiang Ren

Introduction

Fine-grained Entity Typing (FET) is the task of assigning specific semantic types (e.g., /organization/educational_institution) to entity mentions within a given text. To overcome the high cost of manual labeling, researchers typically rely on Distant Supervision (DS), which automatically labels mentions by linking them to external Knowledge Bases (KBs). However, DS is inherently noisy; a mention like "Washington" might be labeled as both a person and a location regardless of context, simply because both exist in the KB.

This noise creates a significant dataset shift problem, where the distribution of the noisy training data differs drastically from the clean test data. While previous work has focused on label denoising, this paper argues that the problem is also architectural: models trained on noisy data often converge to "sharp minima" that fail to generalize. We propose a Masked Virtual Adversarial Training (VAT) framework to regularize the model toward "flat minima," enhancing robustness against distribution shifts.

The Challenge of Dataset Shift and Sharp Minima

The core difficulty in FET with distant supervision is that models easily overfit the noise in the training set. In the loss landscape, this manifests as the model settling into a "sharp minimum." As illustrated in Figure 1, a sharp minimum represents a solution that achieves low loss on the training set but is highly sensitive to small changes in the data distribution ($\Delta s$). When the model is evaluated on clean test data, even a minor shift results in a massive spike in loss ($\Delta L_S$).

Illustration of flat vs. sharp minima under dataset shift.

By contrast, a "flat minimum" maintains low loss over a wider range of the parameter space. Even with a distribution shift, the increase in test loss ($\Delta L_F$) remains minimal. Our goal is to employ regularization techniques that steer the optimization process toward these flatter, more generalizable regions.

Masked Virtual Adversarial Training (VAT) Framework

To achieve flatter minima, we introduce a three-stage framework that combines feature extraction, sample selection, and a novel masked adversarial objective.

1. Feature Extraction

The framework is model-agnostic and can be applied to standard FET architectures such as NFETC (Neural Fine-grained Entity Type Classification) or BERT. These models transform input text into dense vector representations ($Z$).

2. Self-Paced Sample Selection

Not all distantly supervised samples are equally beneficial. We employ a self-paced learning strategy to select training samples. Early in training, the model focuses on samples it is confident about; as training progresses, it incorporates more complex or ambiguous samples. This prevents the adversarial training process from being derailed by extremely low-quality noise early on.

3. Masked VAT

Virtual Adversarial Training (VAT) works by adding small, calculated perturbations to the input features and penalizing the model if its output distribution changes significantly. This encourages local smoothness in the feature space.

The proposed Masked Virtual Adversarial Training framework.

However, standard VAT can be counterproductive in FET if it smooths predictions toward types that are impossible for a given entity. We introduce Masked VAT, which restricts the adversarial search space. The model only considers the candidate types provided by the (albeit noisy) distant supervision. This "masking" ensures that the regularization strengthens the model's consistency among plausible types rather than forcing it to consider irrelevant ones.

Experimental Evaluation

We evaluated our framework on two standard FET benchmarks: Ontonotes (89 types) and BBN (47 types). The training sets for these benchmarks contain approximately 25-30% noise.

  • Performance Gains: The Masked VAT approach achieved state-of-the-art results, improving Strict Accuracy by an average of 3.8% and Macro F1 by 2.5% across datasets.
  • BERT Integration: When applied to a BERT-based backbone (BERT-VAT), the method significantly outperformed the standard BERT baseline, demonstrating that even powerful pre-trained models require specific regularization to handle dataset shift in FET.
  • Feature Space Robustness: T-SNE visualizations confirmed that the VAT-regularized models produced more cohesive clusters in the feature space, indicating that the model learned to group entities by their true semantic types rather than overfitting noisy surface patterns.

Conclusion

This work shifts the focus of FET research from purely label-side denoising to model-side regularization. By utilizing Masked Virtual Adversarial Training, we successfully mitigate the dataset shift problem caused by distant supervision. The framework is highly flexible and can be integrated into existing FET pipelines to improve generalization and robustness in the face of noisy real-world data.

Summary updated

Abstract

The recent success of Distant Supervision (DS) brings abundant labeled data for the task of fine-grained entity typing (FET) without human annotation. However, the heuristically generated labels inevitably bring a significant distribution gap, namely dataset shift, between the distantly labeled training set and the manually curated test set. Considerable efforts have been made to alleviate this problem from the label perspective by either intelligently denoising the training labels, or designing noise-aware loss functions. Despite their progress, the dataset shift can hardly be eliminated completely. In this work, complementary to the label perspective, we reconsider this problem from the model perspective: Can we learn a more robust typing model with the existence of dataset shift? To this end, we propose a novel regularization module based on virtual adversarial training (VAT). The proposed approach first uses a self-paced sample selection function to select suitable samples for VAT, then constructs virtual adversarial perturbations based on the selected samples, and finally regularizes the model to be robust to such perturbations. Experiments on two benchmarks demonstrate the effectiveness of the proposed method, with an average 3.8%, 2.5%, and 3.2% improvement in accuracy, Macro F1 and Micro F1 respectively compared to the next best method.
BibTeX
@inproceedings{shi2020alleviate,
  author = {Haochen Shi and Siliang Tang and Xiaotao Gu and Bo Chen and Zhigang Chen and Jian Shao and Xiang Ren},
  title = {{Alleviate Dataset Shift Problem in Fine-grained Entity Typing with Virtual Adversarial Training}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/539},
}
Figure