Towards Personalized Preprocessing Pipeline Search

Diego Martinez, Daochen Zha, Qiaoyu Tan, Xia HuarXiv 2023 · DOI · arXiv

Abstract

Automated Machine Learning (AutoML) systems typically apply uniform preprocessing pipelines to all numerical features, potentially limiting model performance. This paper introduces ClusterP3S (Personalized Preprocessing Pipeline Search via Clustering), a framework that groups features into clusters and searches for optimal preprocessing pipelines at the cluster level. By utilizing reinforcement learning to navigate the high-dimensional search space, ClusterP3S achieves superior accuracy and efficiency compared to traditional heuristic-based approaches.

Introduction

Feature preprocessing—including cleaning, scaling, and encoding—is a critical yet often overlooked component of AutoML. Current state-of-the-art systems like Auto-Sklearn often apply a "one-size-fits-all" strategy, applying the same transformations to every numerical feature. This approach fails to account for the diverse "personalities" of features, such as varying skewness or cardinality. While searching for a unique pipeline for every feature would be ideal, the resulting combinatorial explosion (e.g., $48^{30}$ for 30 features) makes it computationally infeasible.

The ClusterP3S Framework

To address this challenge, ClusterP3S employs a hierarchical search strategy to reduce the search space from $|P|^D$ to $|P|^K$, where $K$ is the number of clusters and $D$ is the total number of features. The process involves three main stages:

  1. Feature Embedding: Each column is treated as a document; term frequencies are calculated and compressed into dense embeddings via an AutoEncoder.
  2. Upper-Level Search (Clustering): A deep clustering network assigns features to one of $K$ clusters. Unlike static algorithms like K-Means, ClusterP3S uses Reinforcement Learning (REINFORCE) to update the clustering network based on final model accuracy, ensuring clusters are optimized for performance.
  3. Lower-Level Search (Pipeline Selection): For each cluster, the system uses random search to select the best preprocessing pipeline from 48 candidates, including various combinations of imputers, encoders, and scalers.
The ClusterP3S architecture illustrating the hierarchical search loops for clustering and pipeline selection.

Experimental Evaluation

The framework was evaluated on eight diverse classification datasets from the OpenML-CC18 benchmark, such as 38-sick (high missing values) and 1049-pc4 (purely numerical). These datasets were selected to test the system's robustness across varying sample sizes and feature types.

Results and Discussion

ClusterP3S demonstrated a significant performance advantage, outperforming heuristic-based pipelines by an average of 3.5% and random clustering by 2.0%. A key insight is that RL-based clustering outperforms standard K-Means because the RL agent learns to group features based on their behavior within the pipeline rather than just their raw statistical distribution.

A case study on the credit-a dataset highlighted the system's ability to find non-intuitive groupings, such as treating low-cardinality numerical features as categorical variables, which led to measurable performance gains. Furthermore, the system is highly efficient; as shown in the efficiency analysis, ClusterP3S surpasses baseline performance within just 15 seconds of search time.

Average accuracy over time, showing ClusterP3S reaching high performance significantly faster than baselines.

Conclusion

ClusterP3S offers a scalable and effective approach to personalized feature preprocessing. By grouping features and optimizing at the cluster level, it overcomes the combinatorial challenges of per-feature search. Future research will explore integrating preprocessing search with simultaneous model selection and hyperparameter tuning, as well as incorporating fairness-aware constraints into the search process.

Summary updated

Abstract

Feature preprocessing, which transforms raw input features into numerical representations, is a crucial step in automated machine learning (AutoML) systems. However, the existing systems often have a very small search space for feature preprocessing with the same preprocessing pipeline applied to all the numerical features. This may result in sub-optimal performance since different datasets often have various feature characteristics, and features within a dataset may also have their own preprocessing preferences. To bridge this gap, we explore personalized preprocessing pipeline search, where the search algorithm is allowed to adopt a different preprocessing pipeline for each feature. This is a challenging task because the search space grows exponentially with more features. To tackle this challenge, we propose ClusterP3S, a novel framework for Personalized Preprocessing Pipeline Search via Clustering. The key idea is to learn feature clusters such that the search space can be significantly reduced by using the same preprocessing pipeline for the features within a cluster. To this end, we propose a hierarchical search strategy to jointly learn the clusters and search for the optimal pipelines, where the upper-level search optimizes the feature clustering to enable better pipelines built upon the clusters, and the lower-level search optimizes the pipeline given a specific cluster assignment. We instantiate this idea with a deep clustering network that is trained with reinforcement learning at the upper level, and random search at the lower level. Experiments on benchmark classification datasets demonstrate the effectiveness of enabling feature-wise preprocessing pipeline search.
BibTeX
@misc{martinez2023towards,
  author = {Diego Martinez and Daochen Zha and Qiaoyu Tan and Xia Hu},
  title = {{Towards Personalized Preprocessing Pipeline Search}},
  year = {2023},
  eprint = {2302.14329},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.14329},
}
Figure