Abstract
Feedback covariate shift (FCS) occurs when a machine learning model's own predictions are used to select future training data, a common scenario in active learning and biomolecular design. While traditional uncertainty quantification methods like Jackknife+ offer high data efficiency, they are computationally prohibitive due to the need for leave-one-out (LOO) retraining. This paper introduces JAWA-FCS, a method that leverages higher-order influence functions to approximate LOO parameters. JAWA-FCS achieves a 28x computational speedup over exact methods while maintaining rigorous coverage guarantees and producing tighter, more informative uncertainty intervals.
Introduction
In many high-stakes applications, such as protein engineering, models are trained on small datasets to predict the "fitness" of sequences. These models then guide the selection of new sequences for synthesis. This process creates a feedback covariate shift (FCS), where the test distribution is fundamentally altered by the model's own selection criteria. This violates the standard independent and identically distributed (i.i.d.) assumptions, making reliable uncertainty quantification (UQ) both essential and difficult.
Existing UQ methods present a difficult trade-off. Full Conformal Prediction and exact Jackknife+ methods are data-efficient but require massive computational resources for retraining. Conversely, Split Conformal Prediction is fast but requires holding out a significant portion of data, which is often unaffordable in data-scarce environments like wet-lab biology.
Methods
The authors propose JAWA-FCS (JAckknife+ Weighted Approximation for FCS) to provide the data efficiency of Jackknife+ without the retraining bottleneck. The core innovation lies in using higher-order Influence Functions (IFs). By utilizing Taylor-series approximations and the inverse Hessian matrix, JAWA-FCS estimates how model parameters would change if a specific training point were removed.
While influence functions are typically used for model interpretability, JAWA-FCS repurposes them as a computational shortcut. The method specifically handles the unique challenges of FCS, where the likelihood-ratio weights used to correct for distribution shift are themselves dependent on the model's LOO parameters. By using memory-efficient 3rd-order IF approximations, the method makes these complex calculations computationally tractable.
Experimental Evaluation
The method was evaluated on real-world biomolecular design tasks using red and blue fluorescence protein datasets. Small neural network regressors were trained on a constrained sample size of 192 sequences to simulate typical low-data scenarios in protein design.
Key Results
- Computational Efficiency: In protein design experiments, JAWA-FCS reduced the computation time from 84 minutes (for exact JAW-FCS) to less than 3 minutes—a roughly 28x speedup.
- Reliable Coverage: As shown in Figure 1, JAWA-FCS successfully maintained the target coverage level of 90% across various shift magnitudes ($\lambda$), ensuring that true fitness values remained within the predicted intervals.
- Superior Interval Quality: Compared to weighted Split CP, JAWA-FCS produced significantly narrower (tighter) predictive intervals. This increased precision allows researchers to make more confident decisions about which protein candidates to synthesize.
- Improved Design Outcomes: Candidates proposed using the JAWA-FCS framework exhibited a higher mean predicted fitness, demonstrating the practical utility of the method in iterative design loops.
Conclusion and Future Work
JAWA-FCS provides a scalable and data-efficient framework for uncertainty quantification under feedback covariate shift. It effectively bridges the gap between computationally expensive exact methods and data-hungry split methods. Future research will focus on scaling the Hessian-based computations for larger deep learning architectures and establishing formal theoretical bounds for the approximation errors introduced by influence functions.
Summary updated