Abstract
Conformal prediction, which makes no distributional assumptions about the data, has emerged as a powerful and reliable approach to uncertainty quantification in practical applications. The nonconformity measure used in conformal prediction quantifies how a test sample differs from the training data and the effectiveness of a conformal prediction interval may depend heavily on the precise measure employed. The impact of this choice has, however, not been widely explored, especially when dealing with limited amounts of data. The primary objective of this study is to evaluate the performance of various nonconformity measures (absolute error-based, normalized absolute error-based, and quantile-based measures) in terms of validity and efficiency when used in inductive conformal prediction. The focus is on small datasets, which is still a common setting in many real-world applications. Using synthetic and real-world data, we assess how different characteristics -- such as dataset size, noise, and dimensionality -- can affect the efficiency of conformal prediction intervals. Our results show that although there are differences, no single nonconformity measure consistently outperforms the others, as the effectiveness of each nonconformity measure is heavily influenced by the specific nature of the data. Additionally, we found that increasing dataset size does not always improve efficiency, suggesting the importance of fine-tuning models and, again, the need to carefully select the nonconformity measure for different applications.
Introduction
In the field of machine learning, point predictions are often insufficient for high-stakes decision-making. Understanding the reliability of a model's output through uncertainty quantification is critical. Conformal Prediction (CP) has gained prominence as a gold-standard framework because it provides mathematically guaranteed prediction intervals without requiring strong distributional assumptions. However, the performance of CP—specifically its efficiency, or the tightness of its intervals—is highly dependent on the choice of the Nonconformity Measure (NCM). While CP is well-studied in data-rich environments, its behavior under data scarcity remains under-explored. This study investigates how different NCMs perform when training data is limited to small samples (100–1000), a common constraint in specialized domains like medical diagnostics or engineering.
Experimental Methodology
The researchers evaluated Inductive Conformal Prediction (ICP) using three primary categories of NCMs:
- Absolute Error-based: A simple approach using the absolute difference between predicted and true values.
- Normalized Absolute Error-based (normNCM): An adaptive approach that scales error by an estimated local variance.
- Quantile-based (CQR): A method that directly estimates distribution bounds using Quantile Regression.
These measures were tested across various models, including Neural Networks, Gaussian Processes (GP), and Gradient Boosting. The evaluation utilized both real-world UCI datasets and synthetic datasets designed to simulate four distinct noise profiles to observe how NCMs handle different data characteristics.
Key Findings and Discussion
Robust Validity vs. Variable Efficiency
A primary finding is that CP’s mathematical coverage guarantee remains robust even under extreme data scarcity. Regardless of the NCM or model used, empirical coverage almost always met the target level. However, efficiency was highly sensitive to the choice of NCM and the underlying noise structure. While coverage is guaranteed early, achieving stable and narrow interval sizes requires more data to reduce the standard error of the efficiency.
The Challenges of Adaptive Measures
While normNCM is designed to be adaptive, it faces a "double training" challenge in small-data regimes. This method requires training one model for the mean and a second for the variance. With limited data, the variance model often fails to generalize, leading to unstable and excessively wide prediction intervals. In 1D experiments, removing a few "failed" runs where variance estimation exploded reduced the standard error of efficiency by 89.32%.
The "Quantile Tax"
Quantile-based NCMs (CQR) are highly effective for asymmetric or non-Gaussian noise. However, they incur a performance "tax" when applied to simple, symmetric noise profiles. In homoscedastic settings, CQR tends to "over-model" the problem, resulting in wider intervals than simple absolute error measures. This suggests that the complexity of the NCM should be matched to the expected complexity of the data noise.
Conclusion
This study demonstrates that while conformal prediction provides reliable uncertainty bounds even with limited data, no single nonconformity measure is universally superior. Efficiency is heavily influenced by the alignment between the NCM and the data's noise characteristics. For practitioners working with small datasets, the results highlight the importance of careful NCM selection and the potential instability of complex adaptive measures when data is scarce. Future work will explore more robust variance estimation techniques and optimal data splitting strategies for calibration in these constrained environments.
Summary updated