Introduction
Automated Machine Learning (AutoML) has revolutionized the way we approach standard classification tasks, enabling the automatic selection and configuration of machine learning pipelines. However, most existing tools are designed for single-label problems, where each instance belongs to exactly one category. In contrast, real-world data is often multi-label in nature—for example, a single image might be tagged with "beach," "sunset," and "vacation" simultaneously. This research, based on the doctoral work of Marcel Wever, addresses the significant challenges of extending AutoML to the domain of Multi-Label Classification (MLC).
The Challenge: Search Space Explosion
The primary hurdle in automating MLC is the "search space explosion." Unlike single-label classification, MLC often involves complex problem transformation techniques. These methods decompose a multi-label task into a series of single-label sub-problems. This introduces a nested hierarchy of decisions:
- Transformation Strategy: Which method (e.g., Binary Relevance, Classifier Chains) should be used to transform the problem?
- Base Learner Selection: Which algorithm should be used for the resulting single-label tasks?
- Hyperparameter Optimization: How should the parameters for both the transformation and the base learners be tuned?
Traditional black-box optimization techniques, such as Bayesian Optimization, often struggle with this high-dimensional, hierarchical complexity. The search space is simply too vast for flat optimization vectors to navigate efficiently.
Proposed Solution: ML-Plan and HTN Planning
To overcome these limitations, this research introduces ML-Plan, an AutoML framework built upon Hierarchical Task Network (HTN) planning.
HTN Planning for ML Pipelines
Instead of treating a machine learning pipeline as a fixed-length vector of parameters, ML-Plan views pipeline construction as a planning problem. It begins with a high-level goal (e.g., "Classify this dataset") and uses a set of domain-specific rules to refine this goal into increasingly specific sub-tasks, such as "Apply PCA preprocessing," followed by "Use Binary Relevance transformation," and finally "Apply a Random Forest classifier."
This approach offers several advantages:
- Natural Representation: It mirrors the hierarchical way human experts design pipelines.
- Flexibility: It can represent pipelines of varying lengths and structures that flat optimizers cannot easily handle.
- Search Efficiency: By using a greedy best-first search strategy combined with random completions to estimate the potential of unfinished pipelines, ML-Plan can quickly identify promising regions of the search space.
Advanced Techniques for Performance and Efficiency
The research further extends the capabilities of ML-Plan through two key innovations: LiBRe and Runtime Prediction.
LiBRe: Label-Wise Selection
A common simplification in MLC is to use the same base learner for every label. However, different labels may have different characteristics. The LiBRe (Label-Wise Selection) approach exploits the flexibility of the Binary Relevance transformation by selecting the optimal base learner for each individual label. This heterogeneous approach significantly improves generalization performance by tailoring the model to the specific needs of each category.
Runtime Prediction and Meta-Learning
AutoML is computationally intensive, and "zombie" pipelines—those that take an excessive amount of time to train without yielding better results—can stall the entire search process. To mitigate this, a meta-learning component was implemented to predict the runtime of candidate pipelines. By pruning pipelines likely to exceed time budgets before they are even executed, ML-Plan can explore a much larger number of candidates within the same timeframe.
Empirical Evaluation
The effectiveness of ML-Plan was validated through extensive experiments using a pool of approximately 70 machine learning algorithms and transformations. The system was integrated with major software backends, including WEKA, scikit-learn, MEKA, and Mulan.
Key findings from the evaluation include:
- Superior Scalability: ML-Plan consistently outperformed state-of-the-art AutoML systems in MLC settings, demonstrating better scalability as problem complexity increased.
- Effectiveness of Heterogeneity: The LiBRe method proved that allowing different models for different labels provides a significant performance boost over "one-size-fits-all" approaches.
- Search Optimization: Greedy search within a structured planning space was found to be more effective than complex global optimizers in these high-dimensional environments.
Conclusion and Future Directions
This research demonstrates that AI planning is a powerful and natural abstraction for Automated Machine Learning in complex domains like multi-label classification. By moving away from pure black-box optimization and embracing hierarchical structures, ML-Plan provides a more robust and efficient way to navigate the vast search spaces of modern machine learning.
Future work will focus on "safe" pruning of the search space using meta-learning to remove suboptimal branches without missing the global optimum. Additionally, there is a growing interest in making AutoML systems more modular, allowing human experts to more easily inject domain knowledge into the planning process.
Summary updated