Context Independent Noise in Distantly Supervised Entity Typing

27 concepts22 papers1 proposal

Distant supervision is a highly effective strategy for scaling up Information Extraction (IE) and Named Entity Recognition (NER). Instead of relying on tedious, manual human annotation, researchers automatically label massive text corpora by intersecting unstructured text with structured knowledge bases (KBs) like Freebase or Wikidata. If the KB states that an entity possesses a certain attribute or type, any sentence containing that entity can be automatically tagged.

However, this convenience introduces a fundamental flaw: knowledge bases contain global facts, whereas sentences contain local contexts.

When a heuristic blindly maps every known attribute of an entity to every single one of its textual mentions, it creates a severe context-label mismatch. This phenomenon is known as context-independent noise.

The Core Conflict: Global Knowledge vs. Local Context

To understand why context-independent noise occurs, consider an entity like Barack Obama. In a comprehensive knowledge base, this entity is mapped to dozens of fine-grained semantic types, including Person, Politician, Author, and Lawyer.

If we encounter the sentence:

"Obama's latest memoir topped the bestseller list this week."

A standard distant supervision heuristic will tag the mention "Obama" with all four labels. However, within this specific sentence, only Author and Person are contextually active. The types Politician and Lawyer are completely unsupported by the local linguistic evidence.

As demonstrated by Shikhar Murty et al. in Finer Grained Entity Typing with TypeNet, a single sentence usually only activates a small subset of an entity's attributes. Applying the entire set of database attributes introduces massive false-positive noise. Abhishek formalizes this in FgER: Fine-Grained Entity Recognition, explaining that models trained on these unverified labels suffer because they are forced to associate literary context features with political or legal labels. This inconsistent supervision signal severely degrades a model's precision.

This problem is not unique to general web text. In specialized domains like molecular biology, Tomoko Ohta et al. observed in Building an Annotated Corpus in the Molecular-Biology Domain that categorizing entities by their functional roles rather than physical properties creates highly unstable training data, because an entity's biological role shifts entirely based on its context. Similarly, Mohammad Golam Sohrab et al. show in BiomedCurator: Data Curation for Biomedical Literature that matching structured biomedical databases directly to literature routinely forces models to learn coincidental, out-of-context associations.

When the label cardinality is high, treating the entire set of database attributes as ground truth misleads the model with weakly correlated or entirely irrelevant signals, a challenge explored by Shan You et al. in Privileged Multi-label Learning. To build high-precision fine-grained entity typing systems, researchers have developed several core mechanisms to decouple global entity facts from local sentence contexts.


Mechanism 1: Hierarchy-Aware and Partial-Label Embeddings

One way to handle context-independent noise is to redesign how the model views the label space. Instead of treating fine-grained entity types as independent, flat categories, models can leverage the hierarchical relationships inherent in entity ontologies to tolerate or isolate noise.

      [Person]          <-- Supported by context (Keep)
       /    \
 [Artist]  [Politician] <-- Unsupported context-independent noise (Penalize/Drop)

In early work, Dani Yogatama et al. proposed a joint embedding approach in Embedding Methods for Fine Grained Entity Type Classification that mapped both text features and hierarchical labels into a shared low-dimensional space using a ranking loss, making the model naturally resilient to noisy labels.

When a sentence provides enough context to verify a broad type (like Person) but lacks the specific evidence to verify a deep type (like Surgeon), forcing a model to predict the specific type introduces noise. To address this:


Mechanism 2: Structural and Corpus-Level Disambiguation

Instead of evaluating each sentence in complete isolation, models can resolve context-label mismatches by looking at structural patterns across the text or the wider corpus.

For example, Lidong Bing et al. observed in Improving Distant Supervision for Information Extraction Using Label Propagation Through Lists that text lists (e.g., "chest pain, weakness, and nausea") inherently contain highly compatible entities. By constructing a graph of entity mentions and propagating labels through these structural lists, their DIEL framework successfully disambiguates polysemous words and filters out false-positive database matches.

[Sentence List] ---> "Chest pain, weakness, and [Nausea]" 
                           |             |
                     (Contextual Alignment Graph)
                           |             |
                     [Symptom]      [Symptom]  ===> Propagates "Symptom" to [Nausea]
                                                    (Filters out unrelated KB facts)

On a corpus-wide scale, Muhammad Ali et al. developed FGET-RR in Fine-Grained Named Entity Typing over Distantly Supervised Data Based on Refined Representations. They utilized an edge-weighted attentive Graph Convolutional Network (GCN) to refine entity representations. By aggregating global, corpus-level contextual clues, the model gains a macro-perspective of how entities behave across different contexts, allowing it to easily spot when an isolated sentence is being assigned an irrelevant, context-independent label.


Mechanism 3: Capitalizing on the Model Memorization Effect

A powerful insight from deep learning theory is that deep neural networks are lazy learners: they prioritize learning simple, clean, recurring patterns before they begin memorizing complex, inconsistent noise.

Training Progress:
Early Stage ----> Model learns clean, frequent patterns (e.g., "memoir" -> Author)
Late Stage  ----> Model begins memorizing noisy mismatches (e.g., "memoir" -> Lawyer)

Several recent frameworks exploit this behavior to dynamically clean training data mid-flight:

  1. Logit-Based Correction: In Automatic Noisy Label Correction for Fine-Grained Entity Typing, Wei Wei et al. track model logits during early training iterations. If the model consistently outputs a low probability for a distant supervision label, it strongly implies that the label is a context-independent false positive. The framework then automatically relabels or drops the sample.
  2. Trainable Label Distributions: Rather than treating distant supervision labels as static ground truths, Dingkun Long et al. introduced NFETC-AR in Learning with Noise: Improving Distantly-Supervised Fine-grained Entity Typing via Automatic Relabeling. They treat the label distributions themselves as trainable parameters that adaptively evolve alongside the model weights, gradually shifting away from noisy KB assignments toward contextually accurate ones.
  3. Co-Prediction Prompts: Minghao Tang et al. refined this in Learning to Correct Noisy Labels for Fine-Grained Entity Typing via Co-Prediction Prompt Tuning by forcing a language model to make two competing predictions for every entity. This tension exposes systematic errors where a label is valid in a general sense but incorrect for the specific instance.
  4. Denoising Frameworks: As Yue Zhang et al. argue in Denoising Enhanced Distantly Supervised Ultrafine Entity Typing, context-aware representations are absolutely vital because simple filtering methods fail to separate rare, legitimate labels from genuine context-independent errors. This sentiment is echoed by Shuzheng Si et al. in SANTA: Separate Strategies for Inaccurate and Incomplete Annotation Noise in Distantly-Supervised Named Entity Recognition, which explicitly separates wrong labels (inaccurate noise caused by polysemy) from missing labels (incomplete noise caused by KB gaps), applying distinct algorithmic corrections to each.

Mechanism 4: Context Validation via Language Models

Perhaps the most intuitive way to verify if a label matches its context is to read it aloud within the sentence. Hao Zhu et al. turned this intuition into an explicit mechanism called Language Model Enhancement (LME) in Put It Back: Entity Typing with Language Model Enhancement.

Instead of analyzing the entity mention in isolation, the LME framework literally "puts the label back" into the sentence, replacing the entity mention with candidate type descriptions.

Original: "Obama's latest memoir topped the bestseller list..."

Substitution A: "The politician's latest memoir topped the bestseller list..." 
               --> Low LM Probability (Context Mismatch)

Substitution B: "The author's latest memoir topped the bestseller list..."      
               --> High LM Probability (Context Match!)

By querying a pre-trained language model to evaluate the perplexity or probability of these modified sentences, the system can determine which fine-grained labels flow naturally with the surrounding linguistic context. If a sentence containing a substituted label yields a low probability, the label is flagged as context-independent noise and filtered out before it can corrupt the training process.

Key Foundational Work

Embedding Methods for Fine Grained Entity Type Classification (2015) — contributed 1 concept elements to this direction.
Improving Distant Supervision for Information Extraction Using Label Propagation Through Lists (2015) — contributed 1 concept elements to this direction.
Building an Annotated Corpus in the Molecular-Biology Domain (2000) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 22 papers
  2. Version 1 · 2026-10-04 · 49 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_distant_supervision_label_noise_68f6f1c7,
  title = {distant supervision label noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distant-supervision-label-noise/rdr_68f6f1c74ee74a56b3b4fbc27dbdcfa5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_distant_supervision_label_noise_68f6f1c7_v2,
  title = {distant supervision label noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/distant-supervision-label-noise/rdr_68f6f1c74ee74a56b3b4fbc27dbdcfa5/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake