TabDAG: Structured Evolutionary Search for Tabular Feature Engineering

Anastasia Volkova, Andrey Kuznetsov, Ivan Oseledets, Valentin KhrulkovarXiv 2026 · arXiv

TL;DR

TabDAG is a new framework that automates feature engineering for tabular data by combining evolutionary search with a structured graph representation. By forcing a Large Language Model (LLM) to propose specific edits to a feature-dependency graph rather than writing free-form code, TabDAG generates more robust, interpretable, and high-performing features while significantly reducing program failures.

What Problem Does This Paper Address?

Feature engineering—the process of creating new input variables from raw data—is often the "secret sauce" that makes machine learning models work on tabular data. However, it’s notoriously difficult to automate.

Traditional automated systems are often limited to simple math (like adding two columns) and don't understand the context of the data. Recent attempts to use LLMs for this task are more "creative" but suffer from a lack of discipline: they often produce code that crashes, references non-existent columns, or accidentally "leaks" information from the target variable into the features, leading to over-optimistic results that fail in the real world.

Methods

The authors introduce TabDAG, which represents a feature-engineering program as a typed Directed Acyclic Graph (DAG).

  • The Genome (The Graph): Each node in the graph is a transformation. Nodes are explicitly typed (numerical, categorical, etc.) and separated into two interfaces: row-wise (simple math/logic) and aggregate (transformations that need to "fit" on training data, like target encoding).
  • Schema-Constrained Mutations: Instead of asking an LLM to "rewrite this program," TabDAG asks for a "diff" (a set of edits). Crucially, the system regenerates a strict schema for every parent program. This means the LLM cannot even propose a reference to a node that doesn't exist—the structure simply won't allow it to be decoded.
  • Evolutionary Search: It uses an algorithm called MAP-Elites to keep a diverse "archive" of programs that aren't just accurate, but also behave differently (e.g., have different sensitivities to input noise).
  • Rigorous Validation: Before any program is fully evaluated, it passes through static checks (acyclicity, type safety) and behavioral "probes" to detect data leakage.
Figure 1: The TabDAG representation showing a feature graph and a schema-constrained mutation.

Datasets

The researchers evaluated TabDAG across 10 diverse tabular datasets covering regression, binary classification, and multiclass classification tasks. These include well-known benchmarks like:

  • UCI/OpenML: House, Airfoil, Protein, Concrete, Miami, Adult, Students.
  • Kaggle: Churn, Higgs, Otto.

They tested these features against three major model families: CatBoost (gradient boosting), TabM (deep learning ensemble), and TabPFN (a tabular foundation model).

Key Insights and Ideas

  • Structure Beats Free-Form: Constraining the LLM to a graph structure is much more efficient than letting it write Python code from scratch. TabDAG had zero "structural failures" (like syntax errors), whereas baselines frequently produced unrunnable code.
  • LLMs Bring Domain Knowledge: Unlike older "brute-force" tools, the LLM-guided search creates features that make semantic sense. For example, in a concrete strength dataset, it might suggest a "water-to-binder ratio"—a known physical principle—rather than just random arithmetic.
  • Less is More: TabDAG tends to build smaller, deeper, and less redundant feature sets than other methods. It focuses on multi-step compositions (features built on other engineered features) rather than just wide arrays of simple variations.
  • The "Zero-Shot" Head Start: Interestingly, a large portion of the performance gain comes from the LLM's very first suggestions. The evolutionary search then "polishes" these ideas to find the best versions.
Figure 5: Analysis of gains over raw features and the composition of engineered feature sets.

Key Results

  • Broad Improvements: TabDAG improved predictive performance over raw inputs in 26 out of 30 dataset-estimator combinations.
  • Top Ranking: TabDAG and its combination with OpenFE (a popular AFE tool) achieved the two best mean ranks among all compared methods.
  • Robustness: In a head-to-head "ablation" study, the constrained graph approach led to significantly fewer failed evaluations (20.1%) compared to free-form Python (27.9%) or whole-graph JSON rewrites (27.1%).
  • Interpretability: Because the transformations are represented as a graph with clear dependencies and rationales, humans can actually inspect and understand why a new feature was created.
Figure 6: Absolute test metrics across 10 datasets and 3 estimators, comparing TabDAG against various baselines.

Limitations and Future Work

The authors identify several areas for further exploration:

  • Search Budget: The current study used a relatively small budget (100 candidates per run). Larger searches might yield even better results but at a higher API cost.
  • Contamination: As with all LLM research, there's a risk the model has "seen" these public datasets during its training, though the authors used semantic controls (anonymizing column names) to show the search still works without "knowing" the dataset.
  • Scalability: While TabDAG works well on datasets with up to ~90 columns, very high-dimensional data might require different strategies to keep the graph manageable.
  • Transferability: While features can sometimes be "borrowed" between different models (e.g., features found for CatBoost working for a Neural Net), this isn't always guaranteed and depends heavily on the model family.

Summary updated

Abstract

Feature engineering can improve tabular prediction, but useful transformations are compositional, data-dependent, and subject to leakage constraints. We present TabDAG, a framework that combines evolutionary search with a graph representation of feature-engineering programs. Nodes implement row-wise or data-fitted transformations; edges encode dependencies among raw variables, intermediate values, and final features. The representation supports interactions, statistical encodings, clustering- and neighborhood-based features, and multi-stage compositions in a single executable program. Mutations are structured edits constrained by a parent-specific schema and checked before evaluation. In our experiments, no structurally invalid program reached evaluation, although proposals could be rejected and valid graphs could still fail at runtime. Across ten regression and classification datasets and three tabular model families, evolved features improve predictive performance over raw inputs. TabDAG and its combination with OpenFE achieve the two best mean ranks among the compared methods; pairwise rank differences between feature-engineering methods do not survive multiple-testing correction. The LLM-based searches use the same backbone and candidate-evaluation budget. A controlled ablation compares free-form Python, whole-graph JSON, and constrained graph edits with the search engine, budget, and backbone held fixed. Constrained edits yield fewer failed evaluations than either rewrite baseline. The resulting graphs expose the transformations and dependencies selected by search, enabling feature-level analysis of the observed gains.
BibTeX
@misc{volkova2026tabdag,
  author = {Anastasia Volkova and Andrey Kuznetsov and Ivan Oseledets and Valentin Khrulkov},
  title = {{TabDAG: Structured Evolutionary Search for Tabular Feature Engineering}},
  year = {2026},
  eprint = {2611.22828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22828},
}
Figure