Abstract
Automated machine learning (AutoML) systems aim to enable training machine learning (ML) models for non-ML experts. A shortcoming of these systems is that when they fail to produce a model with high accuracy, the user has no path to improve the model other than hiring a data scientist or learning ML -- this defeats the purpose of AutoML and limits its adoption. We introduce an interpretable data feedback solution for AutoML. Our solution suggests new data points for the user to label (without requiring a pool of unlabeled data) to improve the model's accuracy. Our solution analyzes how features influence the prediction among all ML models in an AutoML ensemble, and we suggest more data samples from feature ranges that have high variance in such analysis. Our evaluation shows that our solution can improve the accuracy of AutoML by 7-8% and significantly outperforms popular active learning solutions in data efficiency, all the while providing the added benefit of being interpretable.
Introduction
Automated Machine Learning (AutoML) tools like Auto-Sklearn or TPOT are designed to democratize machine learning for domain experts. However, these systems currently lack a robust "feedback loop." If an AutoML run produces a model with insufficient accuracy, non-expert users are often left with no clear path forward. Without ML expertise, they cannot easily determine if the problem lies in the data quality, missing features, or label imbalance.
Existing active learning solutions attempt to solve this by suggesting which data points to label next, but they often rely on technical jargon like uncertainty scores and assume a single, fixed model. This paper introduces an interpretable feedback mechanism that leverages the diversity of AutoML ensembles to identify specific feature ranges where models disagree, guiding users to collect the most impactful new data.
Methodology: ALE-Based Disagreement
The proposed solution is inspired by the Query-by-Committee (QBC) algorithm, which identifies data points where a "committee" of models disagrees. The authors adapt this for AutoML by using the existing ensemble of diverse models as the committee.
To quantify disagreement, the system uses Accumulated Local Effects (ALE) plots. ALE plots describe how a specific feature influences a model's prediction on average. By calculating the variance of these ALE values across all models in the ensemble, the algorithm identifies "high-uncertainty subspaces"—specific ranges of features where the models are most confused.
This approach provides actionable feedback. For example, as shown in the figure above, the system can distinguish between noisy features (like Source Port) and highly predictive ones (like Destination Port). This allows a domain expert to use their knowledge to generate or find new, relevant samples in specific ranges rather than blindly collecting more data.
Key Results and Evaluation
The researchers evaluated their approach on three distinct datasets: a computer networking dataset (Scream vs. Rest), UCI CoverType, and UCI Internet Firewall.
Accuracy and Data Efficiency
On the networking dataset, the ALE-based feedback improved AutoML accuracy by 7-8%, significantly outperforming uniform sampling and standard confidence-based active learning. The method also proved more data-efficient than traditional QBC in several scenarios.
Statistical Significance
Using the Wilcoxon signed-rank test, the authors demonstrated that these improvements were statistically significant (p-values < 0.05) across multiple test sets. The "Cross-ALE" variant, which runs AutoML multiple times to create a more diverse committee, achieved the highest balanced accuracy of 75.0% in the primary evaluation.
Conclusion and Future Work
By leveraging the inherent diversity of AutoML ensembles, this method transforms model uncertainty into interpretable, actionable feedback for domain experts. While currently optimized for continuous features and dependent on ensemble outputs, it provides a significant step toward making AutoML systems truly iterative and accessible. Future work includes extending the theoretical bounds of QBC to ALE-based subspace sampling and improving the handling of categorical features.
Summary updated