Learning from Biased Weak Supervision

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 235 papers49 conceptsfrozen 2026-10-04

Supervised learning typically requires "gold standard" labels—precise, expert-verified categories for every data point. However, as datasets scale to millions of examples, researchers have turned to weak supervision: cheaper, faster, but less complete signals. This includes Complementary-Label Learning (CLL) (knowing what an object is not), Single Positive Multi-Label (SPML) learning (knowing only one of many possible tags), and Point Annotations (a single click on an object).

The mathematical foundation of these methods usually rests on a "Uniformity Assumption": the idea that if a human provides a partial or negative label, they choose it at random from all possible candidates. But as Tongliang Liu, Elijah Cole, and others have demonstrated, humans are not random number generators. We are systematically biased, and if our models don't account for that bias, they learn a warped version of reality.

The Uniformity Trap

To understand why bias matters, consider the "Risk Estimator"—the mathematical tool used to calculate how well a model is performing based on weak labels.

In Complementary-Label Learning for Arbitrary Losses and Models, Takashi Ishida and Masashi Sugiyama showed that you can theoretically train a perfect classifier using only "negative" labels (e.g., "this is not a dog"). The math works if the negative label is chosen uniformly at random from all incorrect classes. If there are 10 classes and an image is a "cat," the annotator must be equally likely to label it "not dog," "not car," or "not tree."

In reality, human behavior is non-uniform. As noted in Learning with Biased Complementary Labels, annotators exhibit cognitive bias. They are more likely to rule out "obvious" or "visually dissimilar" categories. If you show someone a picture of a Siberian Husky, they are far more likely to say "this is not a toaster" than "this is not a Wolf," even though both are technically correct negative labels.

The Anatomy of Selection Bias

Research across different modalities reveals three recurring patterns of human bias:

  1. Salience and Reporting Bias: In multi-label settings, humans don't pick a random tag; they pick the most prominent one. Elijah Cole and Pietro Perona found in Understanding Label Bias in Single Positive Multi-Label Learning that annotators favor large, central objects. This means "background" classes (like "sky" or "pavement") are rarely selected as the single positive label, leading models to believe these classes simply don't exist.
  2. Tagability: In music and audio, Keunwoo Choi and Kyunghyun Cho identified the concept of "tagability." Some attributes are present but rarely labeled because they are hard for humans to describe or notice, creating a "missing label" problem that is systematic rather than random.
  3. Inhomogeneous Distribution: In 3D vision, Zhiyi Pan observed that sparse point annotations are often "clumpy." Humans click on specific regions of an object (like the center of a car) rather than providing a uniform spatial sample, which Point-DETR3D notes can break models expecting geometric uniformity.

Correcting the Risk

If the bias is systematic, it can be modeled. The research direction has shifted from assuming uniformity to explicitly estimating the transition matrix—the probability that a human will provide label $Y_{weak}$ given the true label $Y_{true}$.

Transition Matrices and SCARCE

Tongliang Liu proposed modifying loss functions by incorporating this transition matrix, effectively "re-weighting" the importance of different labels. More recently, Wei Wang and Gang Niu introduced the SCARCE framework in Learning with Complementary Labels Revisited. Instead of relying on a single global assumption, they treat the problem as a series of binary classification tasks (Positive-Unlabeled learning), which is naturally more robust to non-uniform selection.

Ranking over Estimation

Another breakthrough involves moving away from trying to estimate the exact "risk" of a label. In ComRank: Ranking Loss for Multi-Label Complementary Label Learning, Jing-Yi Zhu and Min-Ling Zhang argue that we don't need to know the exact probability of a bias. We only need to ensure that the "complementary" labels (the "nots") are ranked lower than the potential "true" labels. This ranking approach is inherently more stable when human annotators are inconsistent.

Why It Works: Leveraging the Bias

Surprisingly, bias isn't always a hurdle. Tongliang Liu pointed out that biased labels can actually accelerate convergence. Because human bias is non-random, it contains information. If an annotator says "this is not a toaster" for a Siberian Husky, they are implicitly telling the model something about the visual complexity of the image. By explicitly modeling the "how" and "why" of human labeling choices, we can turn human subjectivity from a source of noise into a source of signal.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 20 papers
  2. Version 1 · 2026-10-04 · 35 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_annotator_selection_bias_eef98dc0,
  title = {annotator selection bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/annotator-selection-bias/rdr_eef98dc05873417eadef3bb7b235dd79},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_from_biased_weak_supervision_eef98dc0_v1,
  title = {Learning from Biased Weak Supervision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-from-biased-weak-supervision/rdr_eef98dc05873417eadef3bb7b235dd79/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}