Abstract
Our goal is to assess if AutoML system changes—i.e., to the search space or hyperparameter optimization—will improve the final model's performance on production tasks. However, we cannot test the changes on production tasks. Instead, we only have access to limited descriptors about tasks that our AutoML system previously executed, like the number of data points or features. We also have a set of development tasks to test changes, e.g., sampled from OpenML with no usage constraints. However, the development and production task distributions are different, leading us to pursue changes that only improve development and not production. This paper proposes a method to leverage descriptor information about AutoML production tasks to select a filtered subset of the most relevant development tasks. Empirical studies show that our filtering strategy improves the ability to assess AutoML system changes on holdout tasks with different distributions than development.
Introduction: The Challenge of AutoML Evaluation
AutoML systems are in a state of constant evolution. Developers frequently iterate on hyperparameter optimization (HPO) algorithms or search space configurations to improve performance. Ideally, these changes would be validated against "production tasks"—the actual datasets and problems encountered by users. However, privacy regulations and legal constraints often make raw production data inaccessible to system developers.
As a result, developers rely on "development tasks" from open-source repositories like OpenML. The fundamental issue is distribution shift: open-source benchmarks often differ significantly from proprietary production data in terms of scale, complexity, and modality. A system change that improves performance on a small, balanced open-source dataset might degrade performance on a high-dimensional, imbalanced production task.
Proposed Methodology: Similarity Filtering
To bridge the gap between development and production, we propose a framework that utilizes Task Descriptors. Even when raw data is restricted, metadata such as the number of features, number of data points, or task modality is typically available. We leverage these descriptors to select a "filtered" subset of development tasks that act as a high-fidelity proxy for production performance.
Our approach involves three primary types of similarity filters:
- Task Descriptor Similarity: Uses Euclidean distance between basic metadata (e.g., $\log_{10}$ of rows and columns) to find development tasks that match the "shape" of production tasks.
- Performance Descriptor Similarity: Employs a surrogate model to predict how a task responds to various hyperparameter configurations. Tasks that exhibit similar performance profiles under the same tuning are deemed similar.
- Oracle Similarity: A heuristic baseline that identifies the theoretical upper bound of filtering performance by observing actual system improvements on holdout tasks.
Key Insights and Empirical Results
Our empirical studies on OpenML datasets reveal several critical insights for AutoML benchmarking:
- The "More is Better" Fallacy: Standard benchmarking assumes that more tasks lead to better evaluations. However, we found that if the benchmark distribution does not match the target, adding more tasks can actually lead to incorrect deployment decisions.
- Efficiency through Filtering: Our filtering strategy demonstrated that using a subset of only ~20% of available tasks can provide an evaluation as accurate as using the full set, significantly reducing computational overhead.
- The Power of Simple Metadata: Surprisingly, simple descriptors like the number of data points were highly effective at identifying relevant proxy tasks, often rivaling the performance of more complex surrogate-based filters.
- Quantitative Gains: In scenarios with significant distribution shift, the proposed filtering strategies consistently reduced "Filter Loss"—the error in predicting production improvement—compared to random task selection.
Conclusion and Future Directions
This work demonstrates that strategic task selection using metadata can mitigate the risks of distribution shift in AutoML evaluation. By focusing on a relevant subset of development tasks, developers can make more informed decisions about system upgrades without needing direct access to production data.
Future research will focus on extending this framework beyond binary classification to include regression and multi-class problems, as well as exploring more complex problem complexity measures to further refine the similarity filters.
Summary updated