Automating Data Preparation for Machine Learning Pipelines

329 concepts279 papers2 proposals

Automated Machine Learning (AutoML) has achieved remarkable success in optimizing the backend of the data science lifecycle. Given a clean, well-structured tabular dataset, modern frameworks can autonomously handle hyperparameter tuning, model selection, and neural architecture search with minimal human intervention.

However, this success relies on a massive, often unstated assumption: that an analysis-ready dataset already exists. In practice, real-world data is messy, unstructured, and riddled with anomalies. As noted by Gerardo Vitagliano et al. in A Scoring-based Approach for Data Preparator Suggestion, data preparation and cleaning frequently consume up to 80% of a project's timeline. This creates a severe bottleneck, forcing practitioners to spend the majority of their time writing manual, brittle preprocessing rules before any automated modeling can begin.

To achieve true end-to-end automation, researchers are shifting focus from algorithmic refinement to data-centric automation. This direction aims to build intelligent systems that can autonomously ingest, clean, and structure raw data for downstream machine learning tasks.

The Clean Data Assumption

Traditional AutoML tools treat data preparation as a series of static defaults rather than a dynamic, optimized process. As Tijl De Bie et al. point out in Automating data science, these frameworks assume a clean matrix with a defined target variable already exists, completely neglecting the upfront effort required for task formalization, data discovery, and initial cleaning.

When confronted with raw, "dirty" data—such as unstructured text, inconsistent categorical variables, or missing values—standard automation tools break down. In an extensive evaluation of prominent AutoML tools, Anh Truong et al. demonstrated that the inability to handle raw or complex formats without human intervention remains the primary failure point for autonomous pipelines. This gap creates a "garbage in, garbage out" dilemma: no matter how sophisticated the downstream neural architecture search is, model performance remains fundamentally constrained by the manual preprocessing steps preceding it.

Framing Data Preparation as an Optimization Problem

Why is automating data preparation so difficult? The primary challenge is the astronomical search space. For any given raw dataset, a data scientist must choose from an infinite combination of transformations: how to impute missing values, how to encode categorical features, which outliers to remove, and how to normalize scales.

To automate this process, researchers frame data preparation as a programmatic optimization problem. Instead of relying on static, human-engineered rules, the pipeline itself is learned.

Reinforcement Learning for Pipeline Discovery

One powerful approach is to treat the sequence of cleaning operations as a dynamic policy search. In Active Reinforcement Learning for Data Preparation: Learn2Clean with Human-In-The-Loop, Laure Berti-Equille introduces a framework that uses reinforcement learning to navigate this search space. By treating data cleaning steps as actions and the downstream model performance as the reward signal, the system automatically discovers the optimal sequence of transformations tailored specifically to maximize the final model's accuracy.

Inductive Programming

Another angle is to synthesize data-wrangling programs directly from data examples. Lidia Contreras Ochando et al. propose using general-purpose Inductive Programming (IP) combined with domain-specific background knowledge libraries (such as date or email parsers). By restricting the program synthesizer's search space to relevant domain functions, the system can learn complex, reproducible data transformations from just one or two user examples, outperforming rigid, specialized tools.

Raw Data ──> [ Program Synthesizer ] ──> Automated Transformation Script ──> Clean Data
                    ▲
                    │ (Guided by Domain Knowledge)
             User Examples

Keeping the Expert in the Loop

While fully autonomous cleaning is the ultimate goal, "cleanliness" is often context-dependent and requires domain expertise. For instance, an outlier in a financial dataset could be a data entry error to be deleted, or it could be a critical signal for fraud detection. Therefore, effective automation must balance algorithmic search with human intent.

To minimize human effort while preserving expert control, researchers leverage semi-supervised and interactive learning:

  • Ensemble-Based Error Correction: In Semi-Supervised Data Cleaning with Raha and Baran, Mohammad Mahdavi et al. demonstrate that a system can learn to detect and correct errors across an entire dataset using as few as 20 human-labeled examples. It achieves this by training a classifier over the outputs of thousands of automated, unsupervised "base" cleaning algorithms, effectively transferring knowledge from prior tasks to minimize manual labeling.
  • Interactive Recommendation Engines: Instead of forcing users to write complex transformation code, systems can act as intelligent assistants. The scoring-based recommendation engine proposed by Gerardo Vitagliano et al. ranks and suggests specific data cleaning steps (preparators) to non-technical domain experts, mitigating choice paralysis and accelerating the preparation phase.

This interactive paradigm ensures that direct manipulation edits are not lost or undocumented outside the formal pipeline—a common reproducibility pitfall highlighted by Oliver Kennedy in The Right Tool for the Job: Data-Centric Workflows in Vizier.

Foundation Models and Declarative Systems

The emergence of Large Language Models (LLMs) and unified database architectures has provided powerful new mechanisms for bypassing the data preparation bottleneck.

LLMs as General-Purpose Preprocessors

Traditional data cleaning tools are highly specialized and difficult to maintain across different datasets. Haochen Zhang et al. explore a radically flexible alternative in Large Language Models as Data Preprocessors. By leveraging the semantic understanding and reasoning capabilities of state-of-the-art LLMs via Chain-of-Thought prompting and batching, the authors demonstrate that a single generative model can match or outperform specialized tools across error detection, data imputation, and entity matching simultaneously. Because LLMs understand context, they can infer missing values or detect anomalies based on semantic meaning rather than rigid syntactic rules.

Unified Declarative Systems

Historically, data preparation and model training have been treated as separate, siloed phases executed across disparate tools (e.g., SQL for cleaning, Python for modeling). This separation prevents global optimization.

To resolve this, systems like SystemDS, introduced by Iulian Antonov et al., integrate data integration, cleaning, and machine learning into a single declarative framework. By utilizing a unified optimizing compiler, the system can perform cross-pipeline optimizations—such as automatically reusing intermediate data transformations or pushing down cleaning operations directly into the data storage layer.

Similarly, the concept of Self-Curating Databases challenges the traditional "ingest-first" model by proposing architectures that automatically discover, link, and enrich heterogeneous data sources in real-time during query execution, removing the need for a separate, manual ETL (Extract, Transform, Load) phase.

Towards True End-to-End Automation

The transition from model-centric engineering to data-centric engineering represents a fundamental maturity phase in artificial intelligence. As summarized by Kyu-Young Whang et al. in A Bird's Eye View of the 4th Industrial Revolution Technologies, systematic improvement of data quality yields far greater returns in real-world production environments than continuous algorithmic tweaking.

By automating the "messy front-end" of data science through reinforcement learning, inductive programming, foundation models, and unified declarative architectures, this research direction bridges the gap between raw data ingestion and automated pipeline generation. The result is a more democratic, reproducible, and scalable data science ecosystem where the path from raw data to actionable physical insights is no longer blocked by a manual engineering bottleneck.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 267 papers
  2. Version 1 · 2026-10-04 · 281 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_data_preparation_bottleneck_66034572,
  title = {data preparation bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-preparation-bottleneck/rdr_66034572cb704c919096efdecb09f226},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_data_preparation_bottleneck_66034572_v2,
  title = {data preparation bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-preparation-bottleneck/rdr_66034572cb704c919096efdecb09f226/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake