TL;DR
Active level set estimation (LSE) finds regions where an expensive black-box function crosses a specific threshold, but incorporating prior knowledge can backfire if that prior is inaccurate. This paper introduces Adaptive Prior Level Set Estimation (AP-LSE), a transfer learning framework that smoothly updates the prior mean using observed function discrepancies. As a result, it accelerates level set discovery when the prior is useful while preventing misleading priors from corrupting sample efficiency.
What Problem Does This Paper Address?
In many physical and engineering tasks—such as finding defective areas on solar silicon ingots, mapping network latency, or stress-testing machine learning models—evaluating a system is expensive. The goal of active level set estimation (LSE) is to efficiently classify inputs into superlevel ($f(x) > h$) or sublevel ($f(x) < h$) sets with the fewest function evaluations possible.
A natural way to improve sample efficiency is transfer learning: incorporating prior knowledge $u_p(x)$ from historical runs, domain simulators, or approximate models. However, standard "vanilla" Gaussian Process (GP) transfer learning assumes the prior is uniformly trustworthy. If the prior mismatches the true function in certain regions of the input space, the GP posterior mean remains biased toward the bad prior. This leads acquisition functions astray, forcing the algorithm to waste expensive queries correcting misleading estimates.
Methods
AP-LSE introduces a locally adaptive prior that dynamically reconciles the prior function $u_p(x)$ with the incoming observations $Y_t$:
- Residual Interpolation: At step $t$, the algorithm computes the mismatch between observed data and the prior, $Y_t - u_p(X_t)$. It models this discrepancy via kernel interpolation: $$u_{dt}(x) = k_t(x)^T (K_t + \sigma^2 I)^{-1}(Y_t - u_p(X_t))$$
- Locally Adjusted Prior: An updated prior function is formed as $u_{p,t}(x) = u_p(x) + u_{dt}(x)$. In regions near observed points, $u_{dt}(x)$ absorbs the mismatch, nudging the prior toward the ground truth. In unexplored regions, $u_{dt}(x) \to 0$, naturally reverting back to the base prior $u_p(x)$.
- Adaptive Posterior Mean: The GP posterior mean is recomputed using this adjusted prior: $$\tilde{\mu}_t(x) = u_{p,t}(x) + k_t(x)^T (K_t + \sigma^2 I)^{-1}(Y_t - u_{p,t}(X_t))$$
- Acquisition Agnostic: $\tilde{\mu}_t(x)$ seamlessly drops into popular active LSE acquisition functions, including the Straddle heuristic, C2LSE, and RMILE.
+-------------------------------------------------------------+
| Active Loop: |
| 1. Fit GP kernel on current observations D_t |
| 2. Compute discrepancy u_dt(x) between Y_t and u_p(X_t) |
| 3. Construct adjusted prior: u_{p,t}(x) = u_p(x) + u_dt(x)|
| 4. Update posterior mean \tilde{\mu}_t(x) |
| 5. Query next point via LSE acquisition function |
+-------------------------------------------------------------+
Datasets
The authors evaluated the method across three synthetic benchmarks and three real-world domains:
- Synthetic Benchmarks:
- Bird (2D): Multimodal benchmark with budget $T=150$.
- MC3D (Multi-circle 3D): Higher-dimensional non-linear surface with $T=400$.
- Mishra03 (2D): Discontinuous-like features with $T=250$.
- Controlled mismatch was induced using a shift parameter $\kappa$.
- Real-World Benchmarks:
- Defective Area Identification: Mapping carrier lifetime on solar cell ingots ($161 \times 121$ grid; target threshold $<80$). Source and target data correspond to two physical silicon ingots.
- Network Latency: Global internet ping dataset collected over consecutive days ($223$ servers on Dec 12 vs. $225$ servers on Dec 13, 2023; threshold $<200$ ms).
- Algorithmic Assurance (AA): Identifying worst-case transformation scenarios (rotation, scaling, shear) that cause CNN error on MNIST to exceed $4\%$. Source model was trained on 75% of MNIST data; target model on 100%.
Key Insights and Ideas
- Vanilla GP transfer suffers from "prior pull": In standard GP regression with a non-zero prior mean, the posterior does not strictly pass through observed values because of noise regularization ($\sigma^2 I$). If the prior is wrong, it continuously pulls the posterior mean away from true observations.
- Two-stage residual correction guarantees lower fitting error: The authors prove that the empirical fitting error of AP-LSE is strictly bounded by that of vanilla transfer learning (Theorem 2).
- Faster convergence in Sobolev spaces: When modeled in a Reproducing Kernel Hilbert Space (RKHS) isomorphic to Sobolev spaces, the generalization error bound of AP-LSE decays at a strictly faster rate than unadjusted transfer learning (Lemma 4).
- Functional priors beat raw-data difference modeling: Unlike approaches such as Diff-GP, which attempt to model the cross-task discrepancy function using raw source points, AP-LSE requires only the prior predictor function $u_p(x)$. It avoids failures caused when source-to-target differences are too complex to learn directly.
Key Results
- Superior Early-Stage Sample Efficiency: Across Bird, MC3D, Mishra03, Defective, Network, and AA tasks, AP-LSE attained higher F1 classification scores within the first 20–50 iterations compared to vanilla transfer, Diff-GP, and learning from scratch.
- Robustness Under Severe Mismatch: On the Mishra03 benchmark with varying prior similarities:
- At 88% similarity ($\kappa=0.2$), AP-LSE reached an 80% F1-score in just 24 iterations (Straddle) / 20 iterations (C2LSE), compared to 88 / 78 for Vanilla, 59 / 56 for Diff-GP, and 243 / 298 from scratch.
- At 67% similarity ($\kappa=0.6$), AP-LSE needed 150 iterations, whereas Vanilla required 217, and Diff-GP failed to reach 80% F1 within the budget entirely (marked as NA).
Limitations and Future Work
- Near-Adversarial Priors: When prior similarity drops substantially below 50% ($\kappa=1.0$, 47% overlap), AP-LSE loses its advantage over training from scratch (taking 165–173 iterations vs. 173–177 for scratch). While it does not drastically hurt performance, it cannot extract signal where none exists.
- Theoretical Assumptions: The convergence analysis strictly assumes an RBF kernel and underlying functions residing in specific Sobolev spaces, though the empirical validation used Matérn kernels.
- Cost Scaling: Computing the additional residual adjustment requires inverting the kernel matrix twice per iteration, scaling with $\mathcal{O}(t^3)$, which is standard for GPs but may become a bottleneck for large evaluation budgets. Extending the framework to sparse GP approximations remains an open direction.
Summary updated