Pool-Based Agnostic Experiment Design in Linear Regression

Masashi Sugiyama, Shinichi NakajimaECML/PKDD 2008 · OpenReview

Abstract

We address the problem of batch active learning (or experiment design) in regression scenarios, where the best input points to label are chosen from a ‘pool’ of unlabeled input samples. Existing active learning methods often assume that the model is correctly specified, i.e., the unknown learning target function is included in the model at hand. However, this assumption may not be fulfilled in practice (i.e., agnostic) and then the existing methods do not work well. In this paper, we propose a new active learning method that is robust against model misspecification. Simulations with various benchmark datasets as well as a real application to wafer alignment in semiconductor exposure apparatus illustrate the usefulness of the proposed method.

Introduction

Active Learning (AL) aims to select the most informative data points for labeling, which is critical when data acquisition is costly, such as in biological assays or semiconductor testing. In pool-based AL, an algorithm selects points from a finite set of unlabeled samples.

Historically, AL methods assume the model is "correctly specified"—meaning the true function generating the data can be perfectly represented by the model. However, real-world models are almost always approximations (misspecified). When traditional AL methods are applied to misspecified models, they often perform worse than random sampling. Furthermore, AL inherently creates a covariate shift: the distribution of selected training data differs from the overall test data. If the model is misspecified, this shift introduces severe bias.

Methodology: Robustness through Importance Weighting

To address model misspecification and covariate shift, we utilize Importance-Weighted Least-Squares (IWLS). A major innovation of this work is how we calculate importance weights and select training points without relying on fragile density estimation. In pool-based AL, the algorithm controls the sampling bias, so the importance weight is simply the inverse of the probability used to sample the point.

We extend two population-based AL criteria to the pool-based setting:

  1. Pool-based Full Variance (P-FV$_W$): Evaluates generalization error expected over both training input points and output noise. It yields a closed-form, analytical solution.
  2. Pool-based Conditional Variance (P-CV$_W$): Evaluates error conditionally, based only on the output noise for a specific, fixed set of training points. This is more accurate for real-world "single-trial" scenarios.
Comparison of full expectation (dash-dot line) versus conditional expectations (dotted lines) for different sets of points. The conditional variance provides a more precise measure for a specific set of training samples.

As illustrated in Figure 1, the full expectation (dash-dot line) represents an average over all possible draws, whereas the conditional expectations (dotted lines) reflect the performance for specific sets of points. By optimizing for the conditional expectation, we obtain a finer assessment of performance for the actual points selected.

The Hybrid Solution

Because P-CV$_W$ lacks a closed-form solution, we use the analytical solution of P-FV$_W$ to generate a family of candidate sampling distributions. We then evaluate these candidates using the superior P-CV$_W$ criterion to select the optimal sampling distribution.

Experimental Results

  • Toy Dataset: In a 1D regression task with a misspecified model, P-CV$_W$ remained stable and accurate as misspecification increased, while traditional methods like Q-OPT catastrophically degraded.
  • DELVE Benchmarks: Across 24 regression datasets, P-CV$_W$ consistently outperformed both passive learning and the P-FV$_W$ baseline. Traditional methods proved highly unstable in higher-dimensional (32D) spaces.
  • Semiconductor Wafer Alignment: Applied to a real-world wafer alignment problem, P-CV$_W$ achieved a ~10% reduction in squared error compared to industry-standard heuristics for marker placement.

Conclusion

This work demonstrates that traditional active learning is brittle under model misspecification. By combining importance weighting with a conditional variance criterion, we provide a robust pool-based AL method that excels in agnostic settings. The ability to obtain importance weights "for free" from the sampling process bypasses the curse of dimensionality in density estimation, making the method practical for high-dimensional real-world problems.

Summary updated

Abstract

We address the problem of batch active learning (or experiment design) in regression scenarios, where the best input points to label is chosen from a ‘pool’ of unlabeled input samples. Existing active learning methods often assume that the model is correctly specified, i.e., the unknown learning target function is included in the model at hand. However, this assumption may not be fulfilled in practice (i.e., agnostic) and then the existing methods do not work well. In this paper, we propose a new active learning method that is robust against model misspecification. Simulations with various benchmark datasets as well as a real application to wafer alignment in semiconductor exposure apparatus illustrate the usefulness of the proposed method.
BibTeX
@inproceedings{sugiyama2008pool,
  author = {Masashi Sugiyama and Shinichi Nakajima},
  title = {{Pool-Based Agnostic Experiment Design in Linear Regression}},
  booktitle = {ECML/PKDD},
  year = {2008},
  url = {https://openreview.net/forum?id=rE3P14MU8C},
}
Figure