Abstract
Automated Machine Learning (AutoML) systems typically apply uniform preprocessing pipelines to all numerical features, potentially limiting model performance. This paper introduces ClusterP3S (Personalized Preprocessing Pipeline Search via Clustering), a framework that groups features into clusters and searches for optimal preprocessing pipelines at the cluster level. By utilizing reinforcement learning to navigate the high-dimensional search space, ClusterP3S achieves superior accuracy and efficiency compared to traditional heuristic-based approaches.
Introduction
Feature preprocessing—including cleaning, scaling, and encoding—is a critical yet often overlooked component of AutoML. Current state-of-the-art systems like Auto-Sklearn often apply a "one-size-fits-all" strategy, applying the same transformations to every numerical feature. This approach fails to account for the diverse "personalities" of features, such as varying skewness or cardinality. While searching for a unique pipeline for every feature would be ideal, the resulting combinatorial explosion (e.g., $48^{30}$ for 30 features) makes it computationally infeasible.
The ClusterP3S Framework
To address this challenge, ClusterP3S employs a hierarchical search strategy to reduce the search space from $|P|^D$ to $|P|^K$, where $K$ is the number of clusters and $D$ is the total number of features. The process involves three main stages:
- Feature Embedding: Each column is treated as a document; term frequencies are calculated and compressed into dense embeddings via an AutoEncoder.
- Upper-Level Search (Clustering): A deep clustering network assigns features to one of $K$ clusters. Unlike static algorithms like K-Means, ClusterP3S uses Reinforcement Learning (REINFORCE) to update the clustering network based on final model accuracy, ensuring clusters are optimized for performance.
- Lower-Level Search (Pipeline Selection): For each cluster, the system uses random search to select the best preprocessing pipeline from 48 candidates, including various combinations of imputers, encoders, and scalers.
Experimental Evaluation
The framework was evaluated on eight diverse classification datasets from the OpenML-CC18 benchmark, such as 38-sick (high missing values) and 1049-pc4 (purely numerical). These datasets were selected to test the system's robustness across varying sample sizes and feature types.
Results and Discussion
ClusterP3S demonstrated a significant performance advantage, outperforming heuristic-based pipelines by an average of 3.5% and random clustering by 2.0%. A key insight is that RL-based clustering outperforms standard K-Means because the RL agent learns to group features based on their behavior within the pipeline rather than just their raw statistical distribution.
A case study on the credit-a dataset highlighted the system's ability to find non-intuitive groupings, such as treating low-cardinality numerical features as categorical variables, which led to measurable performance gains. Furthermore, the system is highly efficient; as shown in the efficiency analysis, ClusterP3S surpasses baseline performance within just 15 seconds of search time.
Conclusion
ClusterP3S offers a scalable and effective approach to personalized feature preprocessing. By grouping features and optimizing at the cluster level, it overcomes the combinatorial challenges of per-feature search. Future research will explore integrating preprocessing search with simultaneous model selection and hyperparameter tuning, as well as incorporating fairness-aware constraints into the search process.
Summary updated