In the world of in silico design—whether engineering a new protein to fight a virus or searching for a room-temperature superconductor—researchers rely heavily on oracles. These are regression models trained on experimental data that predict the properties of novel, unbuilt objects.
The fundamental challenge is that design is an act of extrapolation. We want to find objects better than anything in our training set. However, as design algorithms venture into these uncharted territories, the oracle’s predictions often degrade. In their NeurIPS 2020 paper, Clara Fannjiang and Jennifer Listgarten introduce Autofocusing, a principled way to retrain an oracle on-the-fly, ensuring it remains accurate exactly where the design algorithm is currently searching.
The Problem: Covariate Shift in Model-Based Design
In Model-Based Optimization (MBO), a search model navigates a high-dimensional space, querying an oracle to evaluate candidates because real-world experiments are too slow or expensive.
Typically, oracles are trained once on a fixed dataset and left static. This creates a severe covariate shift: the distribution of points the search algorithm queries becomes vastly different from the distribution the oracle was trained on. As the search progresses toward high-value candidates, the oracle's predictions and uncertainty estimates become untrustworthy. Designers face a catch-22: stay near the known data and discover nothing new, or venture out and be misled by a broken oracle.
The Solution: Autofocusing via a Non-Zero-Sum Game
The authors formalize oracle-based design as a non-zero-sum game between two players:
- The Search Model: Aims to find parameters that maximize the predicted property.
- The Oracle: Aims to minimize the "oracle gap"—the difference between its predictions and the unknown ground truth—specifically in the region the search model is currently exploring.
To solve this without requiring new experimental data, they introduce Autofocusing. The key mechanisms include:
- Alternating Optimization: An "ascent step" updates the search model, followed by a "descent step" that updates the oracle.
- Importance Sampling: Because new labels cannot be collected during the search, the oracle is retrained by re-weighting the existing training data. Data points that resemble the current "region of interest" receive higher weights, while irrelevant points are down-weighted.
- Variance Control: Importance weights can be volatile. The authors use "weight flattening" to balance the bias-variance trade-off, ensuring the oracle doesn't overfit to a tiny subset of the training data.
Experimental Validation
1D Synthetic Task
To illustrate the mechanics of autofocusing, the authors designed a multi-modal 1D function where the training data is intentionally clustered far away from the global maximum.
As shown above, a static oracle gets "stuck" at a local optimum because it cannot reliably extrapolate. In contrast, the autofocused oracle dynamically adjusts its weights, effectively "following" the search model and guiding it successfully to the global peak.
Superconductor Discovery
The method was also evaluated on a real-world dataset of 21,263 materials. To simulate a difficult design task, initial models were trained only on the bottom 80% of materials (by critical temperature). The algorithms were then tasked with discovering high-temperature superconductors in the hidden top 20%.
Testing autofocusing across six different MBO algorithms (including CbAS, RWR, and CMA-ES) yielded impressive results:
- Universal Improvement: Autofocusing improved the performance of all six algorithms across almost every metric, including median and maximum property values.
- Higher Success Rate: For the CbAS algorithm, the chance of finding a material better than anything in the training set jumped from 0.11% to 3.78% with autofocusing.
- Better Extrapolation: Autofocused models achieved significantly higher maximum critical temperatures (e.g., 119.8K vs 103.8K for CbAS), proving they are better at pushing into the tail of the distribution.
Key Insights and Future Directions
Autofocusing provides a mathematical "certificate": if you can minimize the oracle gap in the current search region, your predicted property values are demonstrably closer to the ground truth. It acts as a dynamic covariate shift correction, acknowledging that the target distribution is constantly moving.
While powerful, the method has limitations. It relies on having some training data relevant to the target region; if the search moves into a complete void where all importance weights drop to zero, the method breaks down. Future work may explore more sophisticated variance reduction techniques for the importance weights, dynamic model selection (changing oracle architectures as the effective sample size shrinks), and applications to discrete biological sequences like DNA and proteins.
Summary updated