Reclaiming Latent Information from Noisy Training Data

434 concepts365 papers2 proposals

The standard response to imperfect data is simple: throw it away. If an image is low-resolution, a label is incorrect, or a text pair is misaligned, traditional preprocessing pipelines apply a binary filter. But this "garbage in, garbage out" mentality creates a massive hidden tax: information loss.

When we discard a sample due to surface-level noise, we don't just eliminate the noise; we discard the entire underlying feature representation. An image of a cat mislabeled as a "toaster" still contains all the geometric, textural, and structural features of a cat. A low-resolution image still contains broad semantic layouts. A text document with a few corrupted lines still contains valid syntactic structures.

A growing body of research focuses on moving away from destructive binary filtering toward paradigms that salvage, repair, and exploit the latent information trapped inside noisy data.


The Core Intuition: Feature-Label Orthogonality

Why does keeping noisy data work? The core intuition relies on the fact that data features and data labels are orthogonal properties. A corruption in the label or a localized defect in the feature does not obliterate the remaining signal.

If a model is forced to learn only from pristine data, it suffers from reduced data efficiency and a lack of diversity. By shifting from a binary "keep or discard" mindset to a continuous or multi-stage framework, models can extract robust representations from the underlying distribution of the data, treating noise as a manageable variance rather than a fatal flaw.

Binary Filtering:
[Noisy Sample] ───(Filter)───> [Discarded] (Total Information Loss)

Information Reclamation:
[Noisy Sample] ───(Analysis)───> [Isolate Noise] ───> [Extract Latent Features]

Paradigm 1: Reclaiming Noise as Unlabeled Data

One of the most direct ways to avoid information loss is to strip away untrusted labels while retaining the underlying data points for semi-supervised learning.

Kaijia Yang et al. demonstrate this mechanism in Exploiting Noisy Data in Distant Supervision Relation Classification. Instead of treating noisy instances as garbage to be filtered out, their framework uses reinforcement learning to identify noise and reclaims those instances as unlabeled data within a semi-supervised Variational Autoencoder (VAE). This preserves the latent semantic information and linguistic patterns that would otherwise be lost.

Similarly, Kento Nishi et al. show in Improving Label Noise Robustness with Data Augmentation and Semi-Supervised Learning (Student Abstract) that utilizing the structural information of discarded images for unsupervised or semi-supervised learning dramatically improves data efficiency compared to standard filtering heuristics.


Paradigm 2: Soft Curriculums and Continuous Weighting

Binary filters rely on hard thresholds that are notoriously difficult to tune. A more robust alternative is to ingest all data but dynamically adjust how much the model trusts each sample over time.

In M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis, Fei Zhao et al. introduce a multi-curriculum framework that prioritizes clean data early in training and gradually incorporates noisier examples. This prevents the model from being corrupted early on while ensuring it eventually benefits from the subtle contextual signals present in the lower-quality data.

For generative tasks, Kai Katsumata proposes a soft curriculum in Soft Curriculum for Learning Conditional GANs with Noisy-Labeled and Uncurated Unlabeled Data. Instead of using rigid thresholds to discard low-confidence samples, the framework leverages incorrectly labeled or out-of-distribution samples to improve the model's feature representation, proving that even imperfect data contributes to visual concept learning.


Paradigm 3: Non-Destructive Editing and Masking

When noise is localized—such as a stray word in a sentence or text burned into an image—discarding the entire sample is an overreaction. Non-destructive editing isolates and removes the noise while keeping the rest of the data intact.

In computer vision, Pratyush Maini et al. address this in T-MARS: Improving Visual Representations by Circumventing Text Feature Learning. Traditional vision-language pipelines often discard any image containing text to prevent the model from "cheating." The authors instead mask out the text and re-score the image-text pairs, preventing the loss of high-quality visual data where text and visual content coexist.

In natural language processing, Minh Quang Pham et al. tackle translation divergences in Fixing Translation Divergences in Parallel Corpora for Neural MT. Rather than discarding an entire sentence pair due to minor segmentation errors or stray words, they trim away the non-matching segments. This preserves substantial amounts of valid parallel training information that binary filters would drop.

For instruction tuning, Renhao Li et al. show in CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation that low-quality samples contain useful instructional context that can be salvaged and polished through structured multi-agent editing rather than being filtered out.


Paradigm 4: Label Purification and Latent Normalization

Instead of accepting or rejecting a noisy label, models can actively correct it or map the sample to a clean latent distribution, transforming a liability into an asset.

The authors of Learning to Purify Noisy Labels via Meta Soft Label Corrector, Yichen Wu et al., develop a meta-learning framework that automatically purifies noisy labels into accurate soft labels. This allows the model to retain the valuable structural features of the images without learning erroneous classifications.

From a geometric perspective, Junseok Kwon et al. introduce Wasserstein Distributional Normalization in Wasserstein Distributional Normalization For Robust Distributional Certification of Noisy Labeled Data. Instead of discarding high-loss samples, their method uses Wasserstein gradient flows to mathematically normalize the feature distributions of noisy samples toward those of clean samples, allowing the model to extract representation insights without overfitting to errors.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 355 papers
  2. Version 1 · 2026-10-04 · 351 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_information_loss_in_data_filtering_49371830,
  title = {information loss in data filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-loss-in-data-filtering/rdr_4937183037d64802b681fa8bd8e29259},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_information_loss_in_data_filtering_49371830_v2,
  title = {information loss in data filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-loss-in-data-filtering/rdr_4937183037d64802b681fa8bd8e29259/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake