Entropy Distribution Modeling for Open Set Recognition

This direction is split into other directions: entropy-based novelty detection. Its last version stays readable and citable here.

Version 1 of 1129 papers176 conceptsfrozen 2026-10-04

Standard deep learning models operate under a "closed-set" assumption: they assume every input encountered during deployment belongs to one of the classes seen during training. When faced with something entirely new—a "novel" or "unknown" class—these models typically collapse, assigning high-confidence scores to the most similar-looking known category.

To build robust systems, researchers have turned to Entropy Distribution Modeling. Instead of treating prediction uncertainty as a failure, this direction treats the entropy of a model's output as a primary signal for structural discovery. By modeling the statistical distribution of entropy across a dataset, models can learn to distinguish between what they know and what they don't.

Entropy as a Signal for Novelty

The core intuition is that a well-behaved classifier should be "certain" (low entropy) about familiar data and "uncertain" (high entropy) about unfamiliar data. In MSS: Investigating the Effectiveness of Domain Combinations and Topic Features for Word Sense Disambiguation, Kevin Duh and colleagues demonstrated that high entropy in a classifier's output distribution can identify word senses not found in traditional dictionaries.

However, raw entropy is often noisy. To make it a reliable detector, Dimity Miller and Niko Sünderhauf proposed using Dropout Sampling during inference. By measuring the Shannon entropy across multiple stochastic passes, they found that "model confusion" becomes a quantifiable metric, allowing robots to reject labels for objects they weren't trained to recognize.

Moving Beyond Fixed Thresholds

A common pitfall in early open-set recognition was the use of manual, brittle entropy thresholds. If the entropy is above $X$, call it "unknown." This fails because the "correct" threshold changes depending on the dataset and the model's state.

Modern approaches like UADAL by JoonHo Jang and Byeonghu Na solve this by fitting a Gaussian Mixture Model (GMM) to the entropy distribution of target predictions. By treating the entropy values as a mixture of two populations—low-entropy "knowns" and high-entropy "unknowns"—the model can adaptively estimate the probability that any given sample is novel. Similarly, Zhen Zhang and Bingsheng He utilized a Beta Mixture Model to achieve a similar unsupervised separation in graph-structured data.

Training for Uncertainty

If we want entropy to be a reliable signal, we can't just measure it post-hoc; we must train the model to produce it. This is the logic behind Entropy Regularization.

In PEELER, Bo Liu and Haoxiang Li introduced a loss function that explicitly maximizes posterior entropy for novel inputs during training. By forcing the model to express maximum uncertainty (a uniform distribution) when it encounters something it doesn't recognize, they create a clear "entropy gap." Dan Hendrycks formalized this as Outlier Exposure, where models are exposed to auxiliary "out-of-distribution" datasets and penalized if they assign them high-confidence labels.

Universal and Source-Free Adaptation

This mechanism is particularly powerful in Universal Domain Adaptation (UDA), where a model must adapt to a new domain without knowing if the label sets even match. Kaichao You and Mingsheng Long developed the Universal Adaptation Network, which uses prediction entropy to automatically discover shared categories and reject "unknown" target samples that don't fit the source distribution.

Even in "source-free" scenarios—where the original training data is unavailable for privacy or storage reasons—entropy remains a viable anchor. Pascal Schlachter and Simon Wagner demonstrated that measuring the normalized Shannon entropy of likelihoods from a GMM allows for memory-efficient novelty detection, using only a fraction of the memory required by traditional teacher-student models.

Beyond Simple Classification

The utility of entropy distribution modeling extends to complex tasks like astronomical anomaly detection and semantic segmentation.

  • Astronomy: Aleksandra Ćiprijanović and Ashia Lewis used DeepAstroUDA to push unknown astronomical objects (like merging galaxies) into distinct high-entropy regions, allowing for automated discovery in uncurated sky surveys.
  • Segmentation: In zero-shot segmentation, Dong Un Kang and Se Young Chun showed that pixel-wise entropy can distinguish between foreground objects and background noise without any dense annotations.

By treating uncertainty as a structured distribution rather than a random error, these researchers have moved AI closer to "Epistemic" intelligence—the ability for a model to know when it does not know.

Papers in this version

Showing 60 of 129 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_entropy_distribution_modeling_for_open_s_64a0240e,
  title = {Entropy Distribution Modeling for Open Set Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/entropy-distribution-modeling-for-open-set-recognition/rdr_64a0240e2fac40d789684a6537822947},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_entropy_distribution_modeling_for_open_s_64a0240e_v1,
  title = {Entropy Distribution Modeling for Open Set Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/entropy-distribution-modeling-for-open-set-recognition/rdr_64a0240e2fac40d789684a6537822947/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}