Bag Level Attention for Denoising Distant Supervision

59 concepts49 papers1 proposal

In machine learning, acquiring high-quality labeled data is often the primary bottleneck. To bypass expensive manual annotation, researchers frequently turn to distant supervision or weak labeling. For instance, in relation extraction, if a knowledge base contains the fact founded(Steve_Jobs, Apple), distant supervision automatically labels every sentence in a massive corpus containing both "Steve Jobs" and "Apple" as an instance of the founded relation.

This heuristic creates a fundamental challenge: label noise. While some sentences explicitly state the relation ("Steve Jobs founded Apple in 1976"), others are entirely irrelevant ("Steve Jobs ate an Apple").

To handle this, data is organized into bags. A bag contains all instances (sentences, image patches, or data points) that share a specific weak label. The core problem of Multi-Instance Learning (MIL) is to correctly classify the bag while filtering out the noisy distractors within it.

Why Simple Aggregation Fails

Early approaches to multi-instance learning relied on simple aggregation functions like max-pooling or averaging. As Kai Fan et al. pointed out in Improving Distantly Supervised Relation Extraction with Neural Noise Converter and Conditional Optimal Selector, standard attention mechanisms that average instances tend to drown out the few informative signals within a sea of noise. If a bag contains twenty sentences and only one expresses the target relationship, averaging the representations dilutes the true signal to the point of invisibility.

Conversely, selecting only the single most representative instance—as explored via convolutional networks by Maosong Sun et al. in Incorporating Relation Paths in Neural Relation Extraction—prevents the model from synthesizing complementary information across multiple valid sentences, leading to severe information loss.

Query-Based Selective Attention as a Noise Filter

The breakthrough that addresses this dilemma is selective instance weighting via query-based attention. Instead of treating all instances equally or gambling on a single candidate, the model dynamically scores each instance's relevance.

To understand why this works, consider how the attention weights are computed. The model uses a query—typically a representation of the target relation or the entity pair itself—and computes a similarity score against each instance representation in the bag.

Query (e.g., Target Relation Embedding) 
     │
     ├──► [Similarity Score] ◄── Instance 1 (True Positive)  ──► High Weight (0.9)
     │
     └──► [Similarity Score] ◄── Instance 2 (Noisy Distractor) ──► Low Weight (0.1)

As demonstrated by Linyi Yang et al. in Multi-level Attention-Based Neural Networks for Distant Supervised Relation Extraction, this query-based function enables the model to assign high weights to multiple informative sentences within a single data bag simultaneously. By doing so, the model acts as a soft noise filter. Irrelevant mentions that do not contain evidence for the target labels are effectively ignored, which Yadollah Yaghoobzadeh et al. showed drastically improves precision compared to simple aggregation techniques in Corpus-level Fine-grained Entity Typing.

This mechanism has been refined by leveraging external knowledge graphs. For example, Xu Han et al. in Neural Knowledge Acquisition via Mutual Attention Between Knowledge Graph and Text utilized relation embeddings directly from a knowledge graph as attention queries over text. Because the query embodies the structured semantic definition of the relation, it naturally prioritizes sentences that align with known graph relations and down-weights incorrectly labeled training examples. Similarly, Linmei Hu et al. demonstrated in Improving Distantly-Supervised Relation Extraction with Joint Label Embedding that learning relation labels as semantic embeddings allows for highly precise sentence selection, even in low-resource scenarios where only a few candidate sentences are available.

Beyond Single Vectors: Multi-Aspect and Structured Attention

While a single attention vector helps filter out completely irrelevant instances, relationships in text are often multi-faceted. A single entity pair might share multiple valid contexts across a bag.

To capture this complexity, Jinhua Du et al. introduced a matrix-based selection mechanism in Multi-Level Structured Self-Attentions for Distantly Supervised Relation Extraction. By replacing traditional 1-D attention vectors with 2-D matrices, the model can aggregate information from several relevant instances across different semantic "aspects." This prevents the model from collapsing its focus onto a single best candidate, allowing it to harvest diverse evidence and build a cumulative understanding of the relationship, as seen in the drug-drug interaction models by Shasha Li et al. in their work.

Furthermore, single-author works have shown that incorporating structural and contextual priors can significantly stabilize these weights. Christoph Alt demonstrated in DISTRE that fine-tuning pre-trained language models allows the attention mechanism to leverage deep syntactic and common-sense knowledge to better identify the "long tail" of rare relations. Fenia Christopoulou regularized this selection process in Distantly Supervised Relation Extraction with Sentence Reconstruction and Knowledge Base Priors by forcing sentence representations to align with structured knowledge base link prediction models.

Universality Across Modalities

The power of bag-level selective attention extends far beyond text processing. The exact same mathematical formulation applies whenever models must learn from collections of weakly labeled data across different domains:

  • Computer Vision: When training deep networks on automatically harvested web images, Bohan Zhuang et al. grouped images into bags and used attention to suppress signals from mislabeled distractors in Attend in Groups.
  • Medical Imaging: In whole-slide pathology images, annotations are often coarsely hand-drawn. Zhenzhen Wang et al. developed LC-MIL in Label Cleaning Multiple Instance Learning, using attention weights to identify and prioritize patches that consistently represent the target cancer class, effectively cleaning the rough annotations.
  • Bioinformatics: In single-cell sequencing, data is notoriously sparse and noisy. Sindhura Kommu et al. applied attention mechanisms to weigh individual cell samples in scTransNet, prioritizing informative marker cells over technical outliers to robustly infer gene regulatory networks.

Ultimately, bag-level selective attention succeeds because it transforms the aggregation step from a blind mathematical reduction (like averaging) into an informed, context-aware gating mechanism. By letting the target task guide the filtering process, models can safely extract signal from the massive, messy oceans of weakly supervised data.

Key Foundational Work

Corpus-level Fine-grained Entity Typing (2017) — contributed 2 concept elements to this direction.
Distant Supervision for Relation Extraction with Sentence-Level Attention and Entity Descriptions (2017) — contributed 2 concept elements to this direction.
Neural Relation Extraction with Selective Attention over Instances (2016) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_instance_attention_b1c2ea5f,
  title = {multi-instance attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-instance-attention/rdr_b1c2ea5f7ec5444481e039db01095a56},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_instance_attention_b1c2ea5f_v1,
  title = {multi-instance attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-instance-attention/rdr_b1c2ea5f7ec5444481e039db01095a56/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake