Training a model to recognize relationships between entities—like knowing that "Paris" is the "capital" of "France"—traditionally requires thousands of manually labeled sentences. To bypass this bottleneck, researchers use distant supervision, a technique that automatically generates training data by aligning a structured Knowledge Base (KB) with a large corpus of text.
The core heuristic, popularized by researchers like Ander Intxaurrondo and Eneko Agirre, assumes that if two entities share a relationship in a database, any sentence containing both entities must express that relationship. While this allows for massive scaling, it introduces a fundamental problem: label noise.
The Mintz Assumption and the False Positive Problem
The "Mintz assumption" is the backbone of distant supervision. If Wikidata says (Steve Jobs, founded, Apple), a distantly supervised system will label every sentence containing "Steve Jobs" and "Apple" as a positive example of the founded relation.
As Ludovic Jean‐Louis and Gabor Angeli point out, this heuristic is frequently wrong. A sentence might say, "Steve Jobs once ate an apple," or "Steve Jobs left Apple in 1985." These are false positives—sentences where the entities co-occur for reasons unrelated to the target relationship. In document-level extraction, Chaojun Xiao found that this noise can exceed 60%, creating a significant barrier for accurate model training.
Filtering Noise via Multi-Instance Learning
To handle this, researchers shifted from sentence-level classification to Multi-Instance Learning (MIL). Instead of treating every sentence as a gold-standard label, MIL groups all sentences containing the same entity pair into a "bag." The model then tries to find "at least one" sentence in that bag that actually expresses the relation.
Yujin Yuan and Fei Wu extended this using attention mechanisms. Rather than picking just one sentence, their model assigns weights to every sentence in the bag, allowing the network to focus on high-quality evidence while ignoring noisy, irrelevant sentences. Similarly, Liubo Ouyang used dual-attention mechanisms in Chinese relation extraction to filter out noise caused by incorrect word segmentation and coincidental entity co-occurrence.
Adversarial and Reinforcement Learning for Denoising
If the dataset is a mix of "clean" and "noisy" samples, can we train a second model to act as a filter? This is the intuition behind DSGAN, introduced by Pengda Qin and William Yang Wang. They used a Generative Adversarial Network where a "generator" learns to pick out true positive samples from the noisy pile, while a "discriminator" tries to tell the difference between the generator's choices and actual human-labeled data.
Other researchers, such as Jun Feng and Minlie Huang, framed denoising as a Reinforcement Learning (RL) problem. In their framework, an RL agent acts as an "instance selector" that decides whether to keep or discard a training example based on how much it improves the performance of the main relation classifier. Shun Zheng took this further with DIAG-NRE, which uses RL to discover textual patterns and refines them with a tiny amount of human feedback, significantly boosting precision.
External Validation and Soft Labels
A more recent trend involves using "indirect supervision" from other NLP tasks to verify labels. Kang Zhou and Qi Li introduced DSRE-NLI, which uses a pre-trained Natural Language Inference (NLI) model to check if a sentence actually entails the relationship. For example, if the KB says (A, born_in, B), the NLI model checks if the sentence "A was born in B" is supported by the text.
Instead of hard "correct/incorrect" binary choices, Tianyu Liu proposed soft-labeling. This method allows the model to dynamically adjust the confidence of a label during training. If the model's internal patterns strongly suggest a sentence is a false positive, the soft-label method reduces that sentence's influence on the final weights, preventing the model from overfitting to noise.
Generalizing the Noise Problem
While most work focuses on Relation Extraction, the "distant supervision noise" problem appears across the field:
- Open-Domain QA: Yanfu Xu and Edouard Grave note that assuming any document containing the answer string is "evidence" is a noisy heuristic. A document might contain the word "Paris" without actually explaining why it is the capital of France.
- Named Entity Recognition (NER): Distant supervision using dictionaries often misses entities (false negatives). The Conf-MPU framework addresses this by using confidence scores to ignore unlabeled tokens that are likely entities.
- Entity Typing: Dingkun Long demonstrated that even single-label instances in distantly supervised datasets are often incorrectly categorized, requiring automatic relabeling strategies to prevent the model from learning erroneous type hierarchies.
By moving away from the rigid assumption that "co-occurrence equals relationship," these diverse strategies—from attention and GANs to NLI-based verification—allow models to harvest the scale of the web while filtering out the inevitable noise of automated labeling.