TL;DR
This paper introduces a new statistical framework called Weighted Conformalized Selection (WCS) for reliably identifying promising candidates from a large pool, even when the data distribution shifts between training and testing. WCS offers finite-sample guarantees for controlling the False Discovery Rate (FDR), making it a powerful tool for applications like drug discovery and personalized medicine.
What Problem Does This Paper Address?
In many real-world scenarios, such as drug discovery or healthcare resource allocation, we need to select a subset of "promising" individuals or items from a vast collection. The challenge is to do this reliably, ensuring that the proportion of selected items that aren't actually promising (false discoveries) is controlled. This becomes even harder when the data used for training a predictive model differs from the data we're applying it to – a common issue known as "covariate shift." Traditional methods often struggle with this shift and can lead to unreliable selections. This paper tackles the problem of making reliable selections under covariate shift, without making strong assumptions about the underlying data-generating process or the predictive models used.
Methods
The core of this paper lies in two main innovations:
- Weighted Conformal p-values: The authors introduce a new type of p-value that accounts for covariate shift. These p-values are derived from any predictive model and are "calibrated" to control the probability of a false positive for individual selections. The "weighted" aspect is crucial for handling the distribution shift.
- Weighted Conformalized Selection (WCS): While the weighted conformal p-values are useful for individual error control, combining them for multiple selections is tricky. The authors propose WCS, a novel procedure that leverages these p-values and ideas from multiple testing to control the False Discovery Rate (FDR) in finite samples, even with covariate shift. WCS involves a two-step selection process: an initial selection based on the weighted p-values, followed by a "pruning" step to refine the set of selected candidates.
Datasets
The paper demonstrates the effectiveness of WCS on several real-world datasets:
- HIV screening dataset: Used for drug property prediction, where the goal is to identify molecules likely to bind to a target protein.
- DAVIS dataset: Used for drug-target interaction prediction, focusing on pairs with high binding affinities.
- NSLM observational dataset: Applied to infer individual treatment effects (ITEs) in a semi-synthetic observational setting.
- Bank marketing dataset: Used for outlier detection, where the goal is to identify unusual but potentially important customer contacts.
The paper also mentions the availability of an R package, ConfSelect, that implements WCS and includes code to reproduce the experiments.
Key Insights and Ideas
- Covariate shift is a major hurdle: The paper strongly emphasizes that covariate shift is a prevalent and significant challenge in real-world machine learning applications, often invalidating standard statistical guarantees.
- Conformal p-values need weighting for shift: Standard conformal p-values, which are well-behaved under exchangeability, break down under covariate shift. The proposed "weighted" p-values are essential for maintaining calibration in such scenarios.
- Finite-sample FDR control is achievable: The WCS procedure provides a theoretical guarantee for controlling the FDR in finite samples, which is a stronger guarantee than asymptotic control. This is crucial for applications where making decisions based on a limited number of samples is common.
- Pruning matters: The paper explores different "pruning" strategies within WCS, showing that while they all control FDR, they can have different impacts on power and stability.
Key Results
- FDR Control: Across various applications (drug discovery, ITEs, outlier detection) and under different scenarios of covariate shift, WCS consistently demonstrated empirical FDR control, keeping it below the nominal levels.
- Reliability under Shift: The methods showed reliability in prioritizing candidates even when significant covariate shifts were present between training and test data.
- Comparable Performance: In many scenarios, WCS performed comparably to or better than existing methods like the Benjamini-Hochberg (BH) procedure, especially when dealing with covariate shift.
- Application to ITEs and Outlier Detection: The framework was successfully extended to complex problems like inferring individual treatment effects and detecting outliers, showcasing its versatility.
Limitations and Future Work
- PRDS Violation: The paper highlights that weighted conformal p-values can violate the Positive Regression Dependence on a Subset (PRDS) property, which is a key assumption for standard multiple testing procedures like BH. This necessitates the development of new procedures like WCS.
- Weight Function Estimation: While the theoretical results assume the covariate shift function
w(x)is known, the paper acknowledges that in practice, it might need to be estimated. Theorem 3.5 provides bounds on FDR when using estimated weights, suggesting that the approach is robust to some estimation error. - Rdtm Pruning: The "deterministic pruning" method (Rdtm) was observed to sometimes make too few selections, leading to low power. The authors recommend using "heterogeneous" (Rhete) or "homogeneous" (Rhomo) pruning for better stability and power.
- Unmeasured Confounding: In the context of individual treatment effects, the method relies on the strong ignorability assumption. Handling unmeasured confounding remains an open challenge.
- Computational Cost: While not explicitly detailed as a limitation, the complexity of computing weighted p-values and the multi-step WCS procedure might be a consideration for very large-scale applications.
Summary updated