Confidence Estimation for Semi-Supervised Regression

48 concepts37 papers1 proposal

In semi-supervised learning, the "Softmax Advantage" has historically favored classification. When a classifier predicts a label for an unlabeled image, it doesn't just give a category; it provides a probability distribution. If the model is 99% sure an image is a "cat," that prediction becomes a high-quality pseudo-label for further training.

Regression tasks—predicting continuous values like the age of a person, the depth of a pixel, or the count of a crowd—lack this inherent metric. As noted by Zhi-Hua Zhou et al., continuous outputs provide no built-in confidence score. Without a way to distinguish between a precise prediction and a wild guess, self-training often collapses as the model begins to treat its own noise as ground truth.

The Reliability Gap

The fundamental challenge in semi-supervised regression is identifying which unlabeled samples are "safe" to learn from. In classification, researchers often rely on the cluster assumption: that data points in the same class are separated by low-density regions. However, Arun Raja and Xun Xu argue that these assumptions do not naturally extend to continuous output spaces, where there are no discrete boundaries.

When a model generates a pseudo-label for a continuous variable, any numerical error is directly propagated into the training set. Weihang Dai observes that this lack of consistent uncertainty metrics prevents models from effectively weighting unlabeled samples, leading to the accumulation of errors. To bridge this gap, recent research has focused on three primary strategies: auxiliary critics, probabilistic modeling, and structural reformulation.

Learning to Detect Flaws

One intuitive solution is to train a secondary model whose only job is to judge the primary model's work. Instead of relying on the regressor to know its own limits, researchers introduce a "critic" or "instructor."

For example, Zhanghan Ke et al. proposed using a "flaw detector" for pixel-wise tasks like image denoising. This detector generates a probability map that functions as a proxy for confidence, allowing the system to mask out unreliable pixels before they can corrupt the training process. Similarly, in the context of molecular biology, Fang Wu introduced an "instructor" model to evaluate the reliability of predicted biological effects, effectively vetting pseudo-labels before they are used to augment the training set.

Quantifying Uncertainty via Variance

A more mathematically rigorous approach involves changing what the model outputs. Rather than predicting a single scalar value, these models predict a distribution.

Neal Jean et al. pioneered this by combining deep neural networks with Gaussian processes. By minimizing predictive variance on unlabeled data, their framework allows the model to leverage large datasets even when labels are scarce. Other researchers have turned to Bayesian Neural Networks; Weihang Dai uses variational ensembling to enforce "uncertainty consistency," ensuring that the model's confidence aligns with its actual performance.

In graph-based learning, Yankai Chen et al. model nodes as probability distributions to capture "aleatoric uncertainty"—the noise inherent in the data itself. This allows the model to intelligently weight unlabeled nodes, prioritizing those where the distribution is narrow and the prediction is likely stable.

Reformulating the Problem

If regression is hard because it lacks the discrete structure of classification, why not turn regression into classification?

Pin-Yen Huang et al. developed a framework called RankUp, which converts regression into a ranking-based classification task. By predicting whether one sample's value is greater than another's, the model can utilize standard classification confidence thresholds. A similar logic is applied in 6D object pose estimation by Yichen Zhang et al., who treat continuous geometric regression as a coarse-to-fine classification problem to regain the ability to filter noisy labels.

Alternatively, some methods look for consistency across different "views" of the data. Xiaojun Wan and Xiaojun Wan 0001 demonstrated that in cross-language tasks, if two models trained on different languages agree on a rating, that prediction is highly likely to be reliable. This "agreement" serves as a natural, emergent confidence score in a space where no explicit probability exists.

Key Foundational Work

Semi-Supervised Regression with Co-Training (2005) — contributed 2 concept elements to this direction.
A Mixture of Experts Classifier with Learning Based on Both Labelled and Unlabelled Data (1996) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_regression_confidence_estimation_4d080d06,
  title = {regression confidence estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/regression-confidence-estimation/rdr_4d080d066f6c42f4b1f2518e3501c7fb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_regression_confidence_estimation_4d080d06_v1,
  title = {regression confidence estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/regression-confidence-estimation/rdr_4d080d066f6c42f4b1f2518e3501c7fb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake