Conformal Prediction under Label and Communication Noise

This direction is merged into another direction: conformal prediction with noisy labels. Its last version stays readable and citable here.

Version 1 of 129 papers60 conceptsfrozen 2026-10-04

Conformal prediction (CP) is often celebrated as a "distribution-free" way to get rigorous uncertainty guarantees. If you want a model to output a set of labels that contains the true answer 95% of the time, CP provides a mathematical recipe to achieve exactly that. However, this recipe has a hidden ingredient: it assumes your calibration data—the small dataset used to "tune" the uncertainty—is perfectly labeled.

In the real world, labels are messy. Medical datasets have inter-observer variability, web-scraped data is full of misidentifications, and data transmitted over wireless channels can be corrupted by interference. When the "ground truth" is no longer trustworthy, the statistical guarantees of conformal prediction begin to crumble. Recent research has focused on "noise-aware" conformal frameworks that restore these guarantees by explicitly modeling the corruption process.

The Cost of Unaccounted Noise

Standard conformal prediction works by calculating "non-conformity scores" (how much an image looks unlike its label) on a calibration set and finding a threshold that covers a specific quantile of those scores.

As Matteo Sesia and Y. X. Rachel Wang explain in Adaptive conformal classification with noisy labels, label noise creates a fundamental dilemma for this process. If the calibration labels are wrong, the framework interprets this as model uncertainty. To maintain its mathematical "promise" of coverage, the system becomes overly conservative, producing massive, uninformative prediction sets. In extreme cases, if the noise is systematic rather than random, the model might suffer from "under-coverage," where the true label is missing from the predicted set more often than the user-defined error rate allows.

De-biasing the Threshold

The core intuition behind making CP robust is that we can "de-bias" the calibration threshold if we have even an approximate model of the noise.

In Adaptive conformal classification with noisy labels, the authors introduce a coverage inflation factor. By estimating the divergence between the clean and noisy label distributions—often represented as a transition matrix—they adjust the target error rate. If you know 10% of your labels are likely flipped, you don't aim for 95% coverage on the noisy data; you mathematically shift the threshold to ensure you recover 95% coverage on the underlying clean data.

This idea is extended to large-scale problems by Ethan Fetaya, Coby Penso, and Jacob Goldberger in their work on Noise-Aware Conformal Prediction (NACP). They demonstrate that while standard CP becomes useless as the number of categories grows (like in ImageNet), NACP provides tight, effective coverage guarantees even when the calibration data is heavily contaminated.

Robustness in Real-Time Streams

In online settings, where data arrives in a continuous stream, noise can be even more disruptive. Standard adaptive methods might over-correct for a single noisy label, causing the uncertainty bounds to oscillate wildly.

To solve this, HuaJun Xi, Hongxin Wei, and colleagues proposed the robust pinball loss in Exploring the Noise Robustness of Online Conformal Prediction. This loss function provides an unbiased estimate of the "clean" gradient even when the current label is corrupted. By effectively "ignoring" the noise during the update step, the system achieves a stable convergence rate, ensuring that the reliability of the prediction sets isn't sacrificed for real-time speed.

Uncertainty over Noisy Channels

The problem of noise isn't limited to human labeling errors; it also appears in the infrastructure of distributed AI. In Federated Learning, multiple devices collaborate to train a model. When these devices send their calibration data to a central server over wireless links, the signal itself can be corrupted.

Matteo Zecchin and Sangwoo Park addressed this in Federated Inference with Reliable Uncertainty Quantification over Wireless Channels via Conformal Prediction. They developed a protocol that applies a correction factor based on the signal-to-noise ratio (SNR) of the communication channel. By treating the channel noise as a measurable statistical perturbation, they can adjust the quantile estimation to ensure that the final prediction sets remain valid, even when the data used to build them was transmitted over a "noisy" connection.

From Absolute Labels to Probabilistic Truths

Perhaps the most flexible approach to noise is to stop treating calibration labels as absolute "ground truth."

In Estimating Uncertainty in Multimodal Foundation Models using Public Internet Data, Shiladitya Dutta and Ahmed Alaa introduce WebCP. Instead of relying on a fixed, labeled dataset, they use web-scraped data as an "on-the-fly" calibration set. Because web data is notoriously messy, they assign "plausibility scores" to the labels. By incorporating the uncertainty of the calibration labels themselves into the conformal scoring function, they can provide rigorous guarantees for zero-shot models like CLIP without ever needing a "gold-standard" dataset.

Similarly, Alireza Javanmardi and David Stutz explore Conformalized Credal Set Predictors, which use "first-order" training data (labels as distributions). This allows the model to distinguish between aleatoric uncertainty (inherent randomness in the data) and epistemic uncertainty (the model's lack of knowledge), maintaining coverage even when the calibration data consists of noisy approximations.

Conclusion

The shift toward noise-robust conformal prediction marks a transition from theoretical elegance to practical utility. By moving away from the assumption of perfect data, researchers like Matteo Sesia, Hongxin Wei, and Charles Lu are ensuring that "mathematical guarantees" actually hold up in the messy, noisy environments of medical diagnostics, autonomous driving, and large-scale web AI. The intuition is simple: you don't need perfect labels to get perfect guarantees; you just need to account for the imperfection.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_conformal_prediction_under_label_and_com_67729f5e,
  title = {Conformal Prediction under Label and Communication Noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/conformal-prediction-under-label-and-communication-noise/rdr_67729f5ebdd1480dab08cc935aa95943},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_conformal_prediction_under_label_and_com_67729f5e_v1,
  title = {Conformal Prediction under Label and Communication Noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/conformal-prediction-under-label-and-communication-noise/rdr_67729f5ebdd1480dab08cc935aa95943/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}