Introduction
Fine-grained Entity Typing (FET) is the task of assigning specific semantic types (e.g., /organization/educational_institution) to entity mentions within a given text. To overcome the high cost of manual labeling, researchers typically rely on Distant Supervision (DS), which automatically labels mentions by linking them to external Knowledge Bases (KBs). However, DS is inherently noisy; a mention like "Washington" might be labeled as both a person and a location regardless of context, simply because both exist in the KB.
This noise creates a significant dataset shift problem, where the distribution of the noisy training data differs drastically from the clean test data. While previous work has focused on label denoising, this paper argues that the problem is also architectural: models trained on noisy data often converge to "sharp minima" that fail to generalize. We propose a Masked Virtual Adversarial Training (VAT) framework to regularize the model toward "flat minima," enhancing robustness against distribution shifts.
The Challenge of Dataset Shift and Sharp Minima
The core difficulty in FET with distant supervision is that models easily overfit the noise in the training set. In the loss landscape, this manifests as the model settling into a "sharp minimum." As illustrated in Figure 1, a sharp minimum represents a solution that achieves low loss on the training set but is highly sensitive to small changes in the data distribution ($\Delta s$). When the model is evaluated on clean test data, even a minor shift results in a massive spike in loss ($\Delta L_S$).
By contrast, a "flat minimum" maintains low loss over a wider range of the parameter space. Even with a distribution shift, the increase in test loss ($\Delta L_F$) remains minimal. Our goal is to employ regularization techniques that steer the optimization process toward these flatter, more generalizable regions.
Masked Virtual Adversarial Training (VAT) Framework
To achieve flatter minima, we introduce a three-stage framework that combines feature extraction, sample selection, and a novel masked adversarial objective.
1. Feature Extraction
The framework is model-agnostic and can be applied to standard FET architectures such as NFETC (Neural Fine-grained Entity Type Classification) or BERT. These models transform input text into dense vector representations ($Z$).
2. Self-Paced Sample Selection
Not all distantly supervised samples are equally beneficial. We employ a self-paced learning strategy to select training samples. Early in training, the model focuses on samples it is confident about; as training progresses, it incorporates more complex or ambiguous samples. This prevents the adversarial training process from being derailed by extremely low-quality noise early on.
3. Masked VAT
Virtual Adversarial Training (VAT) works by adding small, calculated perturbations to the input features and penalizing the model if its output distribution changes significantly. This encourages local smoothness in the feature space.
However, standard VAT can be counterproductive in FET if it smooths predictions toward types that are impossible for a given entity. We introduce Masked VAT, which restricts the adversarial search space. The model only considers the candidate types provided by the (albeit noisy) distant supervision. This "masking" ensures that the regularization strengthens the model's consistency among plausible types rather than forcing it to consider irrelevant ones.
Experimental Evaluation
We evaluated our framework on two standard FET benchmarks: Ontonotes (89 types) and BBN (47 types). The training sets for these benchmarks contain approximately 25-30% noise.
- Performance Gains: The Masked VAT approach achieved state-of-the-art results, improving Strict Accuracy by an average of 3.8% and Macro F1 by 2.5% across datasets.
- BERT Integration: When applied to a BERT-based backbone (BERT-VAT), the method significantly outperformed the standard BERT baseline, demonstrating that even powerful pre-trained models require specific regularization to handle dataset shift in FET.
- Feature Space Robustness: T-SNE visualizations confirmed that the VAT-regularized models produced more cohesive clusters in the feature space, indicating that the model learned to group entities by their true semantic types rather than overfitting noisy surface patterns.
Conclusion
This work shifts the focus of FET research from purely label-side denoising to model-side regularization. By utilizing Masked Virtual Adversarial Training, we successfully mitigate the dataset shift problem caused by distant supervision. The framework is highly flexible and can be integrated into existing FET pipelines to improve generalization and robustness in the face of noisy real-world data.
Summary updated