Standard machine learning models are notoriously overconfident, often providing a single "best guess" even when they are uncertain. Conformal Prediction (CP) solves this by producing a prediction set—a collection of possible labels—that is mathematically guaranteed to contain the true label with a user-defined probability (e.g., 95%).
However, these guarantees rely on a fundamental assumption: exchangeability. This assumes that the data the model sees during calibration is statistically identical to the data it will see at test time. Adversarial attacks—whether they are evasion attacks (tweaking test inputs) or poisoning attacks (corrupting calibration data)—intentionally break this assumption. When exchangeability vanishes, the "safety net" of conformal prediction collapses, leading to sets that either exclude the truth or become so large they are useless.
Recent research has focused on restoring these guarantees, moving from simple statistical coverage to certifiably robust uncertainty quantification.
The Exchangeability Gap
In standard CP, we calculate a "non-conformity score" for a calibration set and find a threshold that covers 95% of the data. At test time, we include any label whose score falls below that threshold.
As noted by Ge Yan, Yaniv Romano, and Tsui-Wei Weng in Provably Robust Conformal Prediction with Improved Efficiency, an adversary can nudge a test input just enough to push its non-conformity score above the threshold, causing the true label to drop out of the set. Conversely, as Ziquan Liu and colleagues show in The Pitfalls and Promise of Conformal Inference Under Adversarial Attacks, attacks often force the model to be so uncertain that the prediction set expands to include every possible class, rendering the output uninformative.
Strategy 1: Smoothing and CDF Awareness
One way to resist small perturbations is randomized smoothing. Instead of scoring a single point, we score the average behavior of the model under random noise.
Soroush H. Zargarbashi, Mohammad Sadegh Akhondzadeh, and Aleksandar Bojchevski take this further with CAS (CDF-Aware smoothed prediction Sets). In Robust Yet Efficient Conformal Prediction Sets, they argue that looking only at the mean score is insufficient. By leveraging the full Cumulative Distribution Function (CDF) of smoothed scores, they can produce tighter, more efficient sets that remain robust against both evasion and poisoning.
Strategy 2: Bounding the Network's Sensitivity
If we can mathematically limit how much a model's output changes when the input changes, we can "pre-calculate" the worst-case impact of an attack.
- Lipschitz Bounds: Thomas Massena and his team introduced lip-rcp in Efficient Robust Conformal Prediction via Lipschitz-Bounded Networks. By using networks with a known "speed limit" (Lipschitz constant), they can provide guarantees in constant time, making robust CP viable for real-time applications.
- Neural Verification: In Verifiably Robust Conformal Prediction, Linus Jeary and co-authors use formal verification techniques to ensure that for any perturbation within an $\ell_p$-norm ball, the prediction set will still contain the ground truth.
Strategy 3: Efficiency through Binarization
A major hurdle for robust CP is computational cost. Many methods require hundreds of forward passes to "smooth" the prediction.
To solve this, Zargarbashi and Bojchevski developed BinCP (Robust Conformal Prediction with a Single Binary Certificate). By binarizing model outputs, they reduce the problem to a single binary certificate, achieving state-of-the-art robustness with 100x fewer samples. Similarly, their work on RCP1 (One Sample is Enough to Make Conformal Prediction Robust) demonstrates that robust uncertainty quantification is possible with just a single forward pass, unlocking its use for large-scale models.
Strategy 4: Defending the Calibration Set
While most research focuses on test-time attacks, data poisoning targets the calibration phase. If an attacker modifies the calibration data, the threshold itself becomes biased.
In Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning, Yan Scholten and Stephan Günnemann introduce Reliable Conformal Prediction (RPS). Their approach partitions the data and uses majority-vote aggregation. This ensures that even if a portion of the calibration data is malicious, the resulting uncertainty estimates remain trustworthy.
Summary of the Intuition
Robust Conformal Prediction works by replacing the fragile assumption of "perfect data" with a "worst-case" buffer. Whether through randomized smoothing (averaging out the attack), Lipschitz bounds (limiting the attack's reach), or data partitioning (isolating the attack's influence), these methods ensure that the prediction set remains a reliable safety net, even when an adversary is actively trying to tear it down.