Mitigating False Negative Noise in Distant Supervision

100 concepts78 papers1 proposal

Distant supervision allows researchers to train Information Extraction (IE) models without the prohibitive cost of manual labeling. By aligning external knowledge bases (KBs) or dictionaries with raw text, one can automatically generate "silver-standard" training data. However, this process relies on a dangerous assumption: that the reference KB is exhaustive.

In reality, KBs are inherently incomplete. When a valid entity or relation appears in the text but is missing from the dictionary, distant supervision incorrectly labels it as a negative example. This "false negative noise" forces models to learn that legitimate entities are actually background noise, leading to a significant drop in recall and a bias toward overly conservative predictions.

The Incompleteness Trap

Standard supervised learning operates under a "closed-world assumption," where any token not labeled as an entity is treated as a hard negative. As Minlong Peng et al. argue, this assumption is catastrophic for distantly supervised Named Entity Recognition (DS-NER). Because dictionaries cannot capture every possible entity—especially in informal or evolving domains—the model is actively penalized for correctly identifying novel entities.

This problem extends to relation extraction. Chenhao Xie et al. note that incomplete KBs create a massive class imbalance, where valid relational expressions are drowned out by mislabeled negative samples. This systematic error even corrupts evaluation; Pengshuai Li et al. demonstrate that standard test sets often contain false negatives that unfairly penalize high-performing models for making accurate predictions.

Reframing NER as Positive-Unlabeled Learning

To break the closed-world assumption, several researchers reframe the task as Positive-Unlabeled (PU) learning. Instead of treating unlabeled tokens as negatives, PU learning treats them as a mixture of true negatives and "hidden" positives.

Minlong Peng et al. proposed the AdaPU algorithm, which treats dictionary matches as positive and everything else as unlabeled, allowing the model to achieve performance comparable to fully supervised systems. Building on this, Kang Zhou et al. introduced Conf-MPU, which uses token-level confidence scores to weight the unlabeled data, effectively "ignoring" potential false negatives during the loss calculation. Ken Yano et al. further refined this by splitting the task into span detection and classification, using a "spy-based" PU strategy to identify entities missed by incomplete dictionaries.

Partial Annotation and Marginalized Loss

Another approach involves modifying the model's objective function to account for uncertainty. If we cannot be sure a token is a negative, we should not force the model to commit to that label.

Farhad Nooralahzadeh et al. utilized Partial-CRFs, which marginalize over all possible valid label sequences that are consistent with the known (partial) annotations. This allows the model to remain "undecided" about unlabeled spans rather than assuming they are non-entities. Similarly, Kun Liu et al. proposed a calibrated confidence estimation method that allows a CRF to marginalize out untrusted labels, leveraging the observation that neural networks tend to learn clean patterns before they begin to fit the noise.

Denoising through Reinforcement and Self-Training

Rather than changing the loss function, some frameworks attempt to "clean" the dataset by identifying and reclaiming false negatives.

  • Reinforcement Learning: YaoSheng Yang et al. and Kaijia Yang et al. used RL agents to act as filters. These agents learn to recognize which distantly supervised instances are likely to be noisy and either discard them or reclaim them as unlabeled data for semi-supervised learning.
  • Self-Training: Chen Liang et al. introduced BOND, which uses a two-stage process. First, it trains on the noisy distant labels; then, it uses the model's own high-confidence predictions to iteratively refine the labels, closing the gap with human-annotated models.
  • Adversarial Mining: Botao Yu et al. developed False Negative Adversarial Networks (FAN) to mine hidden positives. By using the early-learning properties of neural networks, they identify instances that the model "wants" to label as positive despite a negative distant label, then use adversarial adaptation to align these instances with the positive distribution.

Global Inference and Structural Completion

A final perspective treats the missing labels as a structural problem. If a KB is a sparse graph, we can use the existing connections to predict where the missing links should be.

Deli Zhao et al. framed distantly supervised relation extraction as a matrix completion problem. By assuming the relationship between textual features and entity relations is low-rank, they used convex optimization to fill in the gaps left by incomplete KBs. Xianpei Han et al. took a different structural approach, using Markov Logic Networks to incorporate global constraints—such as relation entailment and argument types—to ensure that the predicted relations are consistent even when the initial supervision is missing.

By moving away from the assumption that "unlabeled equals negative," these diverse methods allow models to leverage the vast scale of distant supervision while remaining resilient to the inevitable gaps in human knowledge.

Key Foundational Work

New York University 2011 System for KBP Slot Filling (2011) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 81 papers
  2. Version 1 · 2026-10-04 · 127 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_knowledge_base_incompleteness_distant_su_17a5b441,
  title = {knowledge base incompleteness (distant supervision noise)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/knowledge-base-incompleteness-distant-supervision-noise/rdr_17a5b441599145d08691b10fd68631c7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_knowledge_base_incompleteness_distant_su_17a5b441_v2,
  title = {knowledge base incompleteness (distant supervision noise)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/knowledge-base-incompleteness-distant-supervision-noise/rdr_17a5b441599145d08691b10fd68631c7/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake