Automating the Data Preparation Bottleneck

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2281 papers302 conceptsfrozen 2026-10-04

In the popular narrative of artificial intelligence, the "work" happens during model training and inference. However, practitioners know the "80/20 rule": 80% of a data scientist's time is spent on the "janitor work" of data preparation—cleaning messy CSVs, merging disparate databases, and engineering features—leaving only 20% for actual analysis.

While Automated Machine Learning (AutoML) has successfully commoditized model selection and hyperparameter tuning, it often assumes the input data is already pristine. As Hüseyin Uzunalioğlu and colleagues point out in Augmented Data Science, this leaves a massive "manual gap" in the automation pipeline. A new wave of research is now closing this gap, moving from automating models to automating the entire data lifecycle.

The Semantic Bridge: Why Rules Failed

Traditional data cleaning relied on rigid, rule-based systems. If a date format changed or a column header was slightly different, the pipeline broke. This fragility exists because traditional tools lack semantic reasoning—they see strings and numbers, not concepts.

Recent work has shifted toward using Large Language Models (LLMs) and text embeddings to provide this missing context. In Empowering Tabular Data Preparation with Language Models, Mengshi Chen explains that LMs succeed where rule-based models fail because they can "understand" the relationship between data points. For example, the ADEPT framework by Paritosh Ramanan treats raw time-series files as text, using LLM embeddings to filter noise and extract features without any manual cleaning Text embedding models can be great data engineers.

Agentic Orchestration: Decomposing the Mess

Data preparation isn't a single task; it's a sequence of decisions. Should I impute this missing value with the mean or the median? Is this outlier a sensor error or a financial anomaly?

To handle this complexity, researchers are deploying multi-agent frameworks that mimic a human data science team.

By breaking the problem into sub-tasks, these agents can "reason" through the preparation steps, providing a level of flexibility that static scripts cannot match.

Cleaning a dataset is an optimization problem with a nearly infinite search space. David Antony Selby and the team behind CleanSurvival address this by using reinforcement learning (Q-learning) to navigate the space of cleaning options, finding optimal pipelines 10x faster than traditional grid searches CleanSurvival: Automated data preprocessing for time-to-event models.

Similarly, MontePrep uses Monte Carlo Tree Search (MCTS) combined with LLMs to synthesize transformation pipelines even when the "target" data format is unknown, which is crucial for privacy-sensitive enterprise environments MontePrep: Monte-Carlo-Driven Automatic Data Preparation.

Domain-Specific Standardization

In specialized fields like genomics or materials science, the bottleneck is often "format hell." Bin Wang proposed the Data Set Description Language (DSDL) to provide a unified way to describe datasets across different modalities, ending the need for custom coordinate transformation scripts DSDL: Data Set Description Language.

In the biomedical space, Prasanth Bathala introduced BioEL to standardize entity linking, allowing researchers to focus on innovation rather than manual data formatting BioEL: A Comprehensive Python Package for Biomedical Entity Linking. These tools act as "ML-ready" hubs, as seen with Foundry-ML in materials science Foundry-ML and InSilico DB in genomics InSilico DB genomic datasets hub.

The Future: Self-Curating Systems

The ultimate goal of this research direction is to move away from the "ingest-then-clean" model. Kavitha Srinivas and colleagues envision Self-Curating Databases that automatically discover, link, and enrich information in real-time during query execution Self-Curating Databases.

By automating the "artful" and tedious parts of data science—as Alain Biem and others proposed in their early work on cognitive automation Towards Cognitive Automation of Data Science—the field is shifting toward a future where the data prepares itself, allowing human experts to return to high-level hypothesis generation and discovery.

Papers in this version

Showing 60 of 281 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 267 papers
  2. Version 1 · 2026-10-04 · 281 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_data_preparation_bottleneck_66034572,
  title = {data preparation bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-preparation-bottleneck/rdr_66034572cb704c919096efdecb09f226},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_automating_the_data_preparation_bottlene_66034572_v1,
  title = {Automating the Data Preparation Bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automating-the-data-preparation-bottleneck/rdr_66034572cb704c919096efdecb09f226/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}