Does Depth Substitute for Feature Engineering in Tree Models?

Decision-tree ensembles such as XGBoost and CatBoost are still the default for spreadsheet-style (tabular) data. Each tree, however, cuts the raw input columns directly and never builds intermediate features. A tree cannot easily learn a quantity like "price per square metre" unless a human adds that column. Recent models stack layers of trees that can be trained by gradient descent, so that later trees split on features computed by earlier ones. These models claim to learn such features automatically. In parallel, careful benchmark studies keep finding two things. Hand-built features matter more than model choice. Many reported architecture gains disappear once every model gets the same tuning effort.

This note proposes one controlled study that answers a narrow question. We hold the tree family fixed, vary only the number of stacked layers, and give every variant the same hyperparameter search budget. Under those conditions, does adding layers recover the gains that expert feature engineering provides, and how does that gain change as the tuning budget grows? The result would tell practitioners whether to spend effort on architecture depth, on search, or on hand-built features.

Core Claim

We hypothesize that the benefit of stacking tree layers is real but conditional. It should appear mainly on datasets where the winning hand-built features were compositional, meaning ratios, products, or aggregates of raw columns. It should shrink toward zero once those expert features are supplied. Because it is a claim about learned features, it should be measured as a curve over tuning budget rather than as a single tuned score.

If this holds, depth is a partial, automatable stand-in for one specific kind of feature engineering. If it fails, meaning the depth gain does not shrink when expert features are added, or does not appear at all once wide single-layer ensembles are matched in capacity, then the field's "deep trees overcome the shallow bottleneck" story is mostly a tuning or capacity artifact.

Why This Is Not Already Done

Several studies sit close to this question, and each leaves out the piece we need.

GRANDE trains hard, axis-aligned tree ensembles end to end with gradient descent. It reports a mean reciprocal rank of 0.702 against CatBoost's 0.570 on 19 OpenML binary tasks. It is a single-layer ensemble, though, so it says nothing about depth. NODE does stack differentiable oblivious trees, and it beats tuned GBDTs on 4 of 6 large datasets. Its comparison mixes depth, tree type, and training procedure, and it does not report how results vary with tuning budget.

The benchmark studies also stop short. Grinsztajn et al. and Gorishniy et al. level the tuning field between trees and neural networks. Neither isolates tree depth. Tschalzev et al. show that expert feature engineering lifts models into the top 1% of Kaggle leaderboards. They never ask whether a deeper model can learn those features itself.

On the measurement side, Wiemerslage et al. supply the sensitivity metrics we reuse, such as Rel@k and the share of best-equivalent runs. Their study shows that tuning reversed a supposed Transformer-over-LSTM advantage, but it covers sequence models only.

What we add is the combination: a single-variable depth ablation, measured over tuning budget, crossed with the expert feature-engineering tier.

Research Program

The model family. We build one tree family with a depth knob. Layer 1 is a GRANDE-style ensemble of hard axis-aligned trees trained with straight-through gradients, as in GradTree and Learning Tree-Based Models with Gradient Descent. For L = 2 or 3, each tree's leaf outputs become new columns, concatenated with the raw inputs. These are fed to the next layer, in the DenseNet-style wiring NODE uses. Keeping the raw inputs means a deeper model can always fall back to the shallow solution. Any loss from depth is therefore an optimization failure, which we will report as such.

Known risk from GradTree. Learning Tree-Based Models with Gradient Descent reports flat gradients when choosing which feature to split on, which forces random restarts. This constrains the design: we expect the problem to worsen with depth. We will therefore count restarts as part of the tuning budget, not as free extras.

The evaluation protocol. It is built on OpenML tasks with fixed, versioned splits. Every model gets the same search space design principle and a pool of 200 random-search trials per dataset, each trial run with three seeds. We use multiple seeds because Locatello et al. showed that seed variance can masquerade as method differences. Hyperparameters are selected on a validation split nested inside the training data, so the test split is never touched during search.

The main measurement: budget curves. From the trial pool we compute expected-best-test-score curves for k trials, k from 1 (defaults) to 200. Each curve is estimated by resampling subsets of trials from the pool. We then plot the depth gain, the curve for L = 2 or 3 minus the curve for L = 1, as a function of k. Alongside it we report Rel@k and best-equivalent-run percentages. We also report everything against wall-clock time, because NODE needed about 7.5 GPU-minutes where CatBoost needed 41 CPU-seconds on YearPrediction. A gain that exists only per trial, and not per hour, is a different finding from one that holds per hour.

The feature-recovery probe. On the Kaggle suite we test whether intermediate layers learn what experts built. For each expert-engineered column from the winning solutions, we fit a small held-out regressor that predicts it from the layer-1 leaf outputs. We compare that predictability with the same regressor fitted on the raw columns. This tests the mechanism directly rather than only the final score.

Baselines

The naive reference: CatBoost at default settings. Default CatBoost on standardized preprocessing, with k = 1 on the budget curve. This is the no-effort practitioner choice, and every curve is reported relative to it.

The strongest existing competitor: tuned CatBoost with expert features. This means CatBoost tuned with the same 200-trial random search and given the expert-feature tier from Tschalzev et al.. It is the hardest competitor because that study found feature engineering to be the dominant factor, and CatBoost also handles categorical columns natively. Beating this baseline would require a stacked-tree model on standardized features to match or exceed CatBoost-with-expert-features. We will test this on a majority of the Kaggle tasks, using paired seed-level tests. No absolute per-dataset number is reported in the file; only the "top 1%" leaderboard result is given, so we will reproduce this baseline ourselves.

The strongest published tabular-tree result: GRANDE on the 19-dataset suite. On the 19 OpenML binary suite we rerun GRANDE (our L = 1 model), CatBoost, XGBoost, and NODE under our protocol. We check against the reported mean reciprocal ranks: 0.702 for GRANDE, 0.570 for CatBoost, 0.395 for NODE. If our rerun ranks disagree with these, that is itself a finding about tuning confounds.

Neural-network reference: a tuned ResNet. A ResNet tuned the same way, since Gorishniy et al. found it a strong deep baseline.

Ablation 1: capacity-matched wide shallow model. An L = 1 model whose tree count and tree depth are tuned within a range that matches the parameter count of L = 3. This separates "more layers" from "more trees."

Ablation 2: generic interaction features. An L = 1 model, and CatBoost, given automatically generated pairwise ratio and product features of the top-ranked raw columns. This separates learned compositional features from cheap brute-force ones.

Why not just…?

Why not just tune CatBoost harder? Grinsztajn et al. found that flat trees win on medium-sized data, so this is a serious alternative. Our budget curves answer it directly. If CatBoost's curve dominates the L = 3 curve at every budget and every preprocessing tier, depth is not worth paying for, and we will say so. The existing evidence that it might not dominate is GRANDE's reported reciprocal-rank lead, which our rerun will either confirm or erode.

Why not just add more trees to a single layer? Any depth gain could simply be extra capacity. This is the most likely way our claim fails, and Ablation 1 exists to test it. Depth must beat the capacity-matched wide model specifically on the datasets whose expert features were compositional. If it does not, the hypothesis is rejected.

Why not just generate ratio and product features automatically and keep shallow trees? This is cheap and may well work. If Ablation 2 matches the stacked models, the honest conclusion is that depth buys nothing beyond brute-force interaction features. The feature-recovery probe then tells us whether stacked layers at least find the specific features experts chose, which brute-force generation would only find by enumeration.

Why not just use a tuned MLP or ResNet, which already has depth? Grinsztajn et al. showed that such networks are hurt by uninformative features and by rotation invariance, a bias that suits tabular data poorly. Stacked axis-aligned trees are meant to keep the tree bias while adding depth. We include the ResNet so that this is checked rather than assumed.

Proposed Experiments and Decisive Checks

Datasets. The 19 OpenML binary suite tests whether results reproduce. The 45-dataset suite of Grinsztajn et al. provides breadth on medium-sized data. The 10 Kaggle tasks of Tschalzev et al., run under both standardized and expert-feature tiers, test the core claim.

Synthetic controls. Two kinds of target come first: targets defined additively on raw columns, and targets that depend on ratios or thresholded products of columns. Depth gain should be near zero on the first and positive on the second. As a sanity check that axis-aligned behaviour survives stacking, we repeat the random-rotation test from Grinsztajn et al. Stacked trees should degrade under rotation the way flat trees do.

First decisive check. The depth gain at k = 200 is compared between the standardized tier and the expert tier on the Kaggle suite. The hypothesis predicts a significantly larger gain under standardized features, measured by paired bootstrap over seeds and datasets.

Second decisive check. The depth gain must be positive against the capacity-matched wide model.

Third decisive check. In the feature-recovery probe, layer-1 leaf outputs must predict expert features better than raw columns do, and better on compositional features than on purely categorical encodings.

Reading the results. Passing all three checks supports depth as automatable feature engineering. Failing the second check means depth is just capacity. Failing the first means depth and feature engineering help independently, which would also be useful to know.