Introduction
Model-based design (MBD) is a cornerstone of modern protein engineering and materials science, where the objective is to discover novel objects with optimized properties. In practice, this involves training a proxy model—an "oracle"—on existing experimental data to predict properties of interest. Optimization algorithms then search the design space for candidates that maximize these predictions. However, a fundamental challenge arises: to find truly superior designs, the search must extrapolate into regions far from the training distribution. In these uncharted territories, the oracle often suffers from covariate shift, leading to unreliable predictions and failed design cycles.
Methodology: Autofocusing as a Non-Zero-Sum Game
To address this, we introduce autofocusing, a technique that dynamically retrains the oracle to maintain accuracy in the specific regions currently being explored by the search model. We frame the design process as a non-zero-sum game between two components:
- The Search Model: Aiming to identify parameters that maximize the predicted property.
- The Oracle: Aiming to minimize the "oracle gap"—the discrepancy between its predictions and the ground truth—within the search model's current distribution.
Implementation is achieved through importance sampling. At each iteration, the search model updates its focus. Simultaneously, the oracle is retrained on the original dataset, but data points are re-weighted based on their proximity to the search model's current area of interest. This "focuses" the oracle's limited capacity where it is most critical for guiding the search.
Experimental Evaluation
The efficacy of autofocusing was evaluated on two primary benchmarks:
- Synthetic 1D Task: A multi-modal function where training data is intentionally biased away from the global maximum. This serves to visualize how autofocusing enables the search model to "cross the valley" of low-performing regions to reach the true peak.
- Superconductivity Discovery: A real-world dataset of over 21,000 materials. To simulate a discovery scenario, the top 20% of high-performing materials were withheld from the initial training set, forcing the model to extrapolate.
Key Results and Insights
The inclusion of autofocusing yielded statistically significant improvements across various model-based optimization (MBO) algorithms. In the superconductivity task, the CbAS algorithm saw its maximum discovered critical temperature rise from 103.8K to 119.8K. Furthermore, the "Percent Chance of Improvement" (PCI)—the likelihood of a suggested design outperforming the best training sample—increased dramatically. For some methods, the PCI jumped from near zero to over 4%, indicating a much higher reliability in finding true peaks.
A critical theoretical insight is the Oracle Gap Certificate. By minimizing the oracle gap, we obtain a mathematical guarantee of the oracle's reliability in the local search region. Autofocusing works synergistically with "trust region" methods, which naturally constrain the search space. This constraint prevents the search model from wandering too far into regions with zero data density, which in turn mitigates the high variance often associated with importance weights, ensuring the retraining process remains stable.
Limitations and Future Directions
While effective, autofocusing relies on importance sampling, which can become unstable if the search model deviates too drastically from the training data. Future research may explore more robust density ratio estimation techniques, such as discriminative classifiers. Additionally, dynamic model selection—adjusting the oracle's architecture or capacity during the search—presents a promising avenue for further enhancing design performance in complex landscapes.
Summary updated