Abstract
This chapter discusses the design of workflows (or pipelines), which represent solutions that involve more than one algorithm. This is motivated by the fact that many tasks require such solutions. This problem is non-trivial, as the number of possible workflows (and their configurations) can be rather large. This chapter discusses various methods that can be used to restrict the design options and thus reduce the size of the configuration space. These include, for instance, ontologies and context-free grammars. Each of these formalisms has its merits and shortcomings. Many platforms have resorted to planning systems that use operators. These can be designed to be in accordance with the given ontologies or grammars. As the search space may be rather large, it is important to leverage prior experience. This topic is addressed in one of the sections, which discusses rankings of plans that have proved to be useful in the past. The workflows/pipelines that have proved successful in the past can be retrieved and used as plans in future tasks. Thus, it is possible to exploit both planning and metalearning.
Introduction
In modern machine learning, the focus has shifted from selecting a single optimal algorithm to designing a comprehensive sequence of operations known as a workflow or pipeline. A typical real-world application involves multiple stages: data cleaning, feature engineering, model selection, and post-processing.
The primary challenge in automating this design is the "curse of dimensionality." Combining dozens of preprocessing techniques with numerous classifiers—each possessing unique hyperparameters—results in an astronomical search space. To manage this complexity, researchers employ "declarative bias" to restrict the search to logically sound sequences.
Methodology for Automated Design
The automation of pipeline design is generally approached through three distinct pillars:
1. Constraining the Search Space
To prevent the exploration of invalid or nonsensical pipelines, formal structures are used to define the "legal" combinations of algorithms.
- Ontologies: These provide a hierarchical categorization of algorithms (e.g., distinguishing between "Decision Trees" and "Naive Bayes"). This allows the system to reason about the types of tools available and their appropriate roles.
- Context-Free Grammars (CFGs): CFGs define formal rules for pipeline construction. For instance, a rule might specify that a "Thresholding" operation must follow a "Probabilistic Classifier," effectively pruning thousands of invalid configurations before the search begins.
2. Search and Planning Strategies
Once the space is constrained, the system must find the optimal path.
- AI Planning: Algorithms are treated as "operators" with specific pre-conditions (e.g., "requires nominal data") and post-conditions. Planners then determine a logical path from raw data to the final objective.
- Evolutionary Algorithms: Tools like TPOT utilize genetic programming to evolve pipelines over generations, recombining successful components from "parent" workflows.
3. Metalearning and Experience Leveraging
Rather than starting from scratch for every new dataset, systems can store a "portfolio" of historically successful workflows. By ranking these based on performance on similar datasets, the system can quickly identify high-quality starting points.
Evaluation and Results
The effectiveness of these methods was evaluated using the OpenML Benchmark Suite, comparing a metalearning-based system (AR) against Auto-WEKA* across 100 diverse datasets.
The results demonstrate that metalearning serves as a significant advantage under tight time constraints. When limited to a 5-minute budget, AR* outperformed Auto-WEKA on 35 datasets, while Auto-WEKA won on only one. As the time budget increases to 60 minutes, the optimization-heavy approach of Auto-WEKA begins to close the gap, eventually surpassing the metalearning approach in more cases.
Conclusion and Future Directions
While automated pipeline design has seen significant progress, several challenges remain. Defining precise "operators" for AI planning is currently a manual, expert-driven process. Furthermore, the initial stages of the KDD process—such as problem formulation and domain understanding—and the final interpretation stages still require substantial human intervention. Future research is moving toward "Automating Data Science" more broadly, aiming to handle complex, non-linear workflows beyond simple linear pipelines.
Summary updated