Many tabular systems turn continuous numbers into a few categories before learning. Tokenized tabular transformers do it, models that read tables as text do it, interpretable rule lists and trees do it, and gradient-boosted trees do it internally with histograms. A common working belief is that enough data will wash out the damage. The clinical and statistical literature points the other way: binning creates blind spots near thresholds and erases variation inside a bin, and no amount of data restores a value the model never sees.
We propose that both views are partly right, and that one measurable quantity decides between them on a given dataset. Data can repair binning only to the extent that the exact position of a value inside its bin can be inferred from the other columns. When it can, a model given enough rows learns to reconstruct the lost detail, and the gap between binned and continuous models shrinks with scale. When it cannot, the gap settles at a fixed floor that no amount of data removes. We will turn this into a scaling law for loss as a function of rows and bin count, test it on real benchmarks, and use it to predict, from small-data runs, how many bins a dataset needs at a given size.
Core Claim
Let $x_j$ be a continuous feature cut into $B$ bins, and let $r_j$ be its offset from the center of its bin. Define $R^2_j$ as the fraction of the variance of $r_j$ that the remaining features can explain, estimated with a held-out regressor on the full data. Our claim is that the excess loss of a sufficiently flexible model trained on binned inputs, relative to the same model on continuous inputs, behaves as
$$\Delta(N,B) \;\approx\; \underbrace{c\sum_j w_j\,(1-R^2_j)\,B^{-\gamma}}_{\text{floor that data cannot remove}} \;+\; \underbrace{c'\sum_j w_j\,R^2_j\,B^{-\gamma}\,N^{-\delta}}_{\text{part data can recover}}.$$
Here $N$ is the number of training rows, $w_j$ is how sensitive the target is to feature $j$ (measured by the continuous model's partial-dependence slope), $\gamma$ is the rate at which finer bins shrink the step-function approximation error, and $\delta$ is the rate at which the model learns to reconstruct within-bin position from correlated columns. For a smooth target and roughly equal-mass bins, piecewise-constant approximation error in squared loss scales with bin width squared, which suggests $\gamma \approx 2$. We will fit $\gamma$ rather than assume it.
Two consequences can be tested. First, the asymptotic floor should track $\sum_j w_j(1-R^2_j)$ across datasets: high-redundancy tables should show the gap shrinking with $N$, and low-redundancy tables should show it flattening. Second, the best bin count $B^*(N)$ grows with $N$, at a rate set by the recoverable term. Classical one-feature histogram rules cannot account for this because they ignore the other columns.
Research Program
The motivating evidence comes from Direction 1. Several studies document that thresholding hides sub-threshold signal or non-monotone structure:
- Regression-based Deep-Learning predicts molecular biomarkers from pathology slides finds this for biomarker dichotomization.
- Creatinine clearance versus serum creatinine as a risk factor in cardiac surgery finds it for renal cutoffs.
- Characterization of non-monotonic relationships between tumor mutational burden and clinical outcomes finds it for single-cutoff binning of mutational burden.
All three work at fixed sample size. None asks whether a larger cohort would let a model route around the cut.
Direction 2 supplies the opposing intuition. Scaling to Very Very Large Corpora for Natural Language Disambiguation found that more data narrows the gaps between algorithms. Generating QM1B with PySCFIPU found that a model without physics priors improves from 285 to 32 meV mean absolute error from data alone.
The closest methodological template is Scaling Laws for Precision. It folds weight bit-width into a scaling law through an "effective parameter count" and shows that more training data can hurt after quantization. We borrow its approach of treating resolution as a scaling axis. Our setting differs in two ways: we discretize the inputs rather than the weights, and our proposed interaction term depends on redundancy among features, which has no analogue for weights.
The work has three parts.
Measurement harness. For each dataset we draw nested subsamples spanning roughly three orders of magnitude in $N$. We bin every numerical column at $B \in \{2, 4, 8, 16, 32, 64, 256\}$ equal-mass bins and also keep the continuous version. We train four model families on each cell:
- Gradient-boosted trees, where the histogram size
max_binis the knob. - An MLP with hard one-hot bins, versus the same MLP with ordered piecewise-linear encoding at the same $B$, following On Embeddings for Numerical Features in Tabular Deep Learning. This isolates whether ordered "fill-level" encoding of bins changes $\gamma$ or the floor.
- A small row transformer with quantile tokens in the style of Tabular Transformers for Modeling Multivariate Time Series.
- As an interpretability regime, depth-limited trees, where we compare binarized splits against the continuous-split method of CLARITree.
Redundancy estimate. For each feature and bin count we compute $R^2_j$ by regressing within-bin offsets on all other features, using held-out folds. This number is computed before any scaling run, so it acts as a genuine predictor rather than a fitted parameter.
Fitting and extrapolation. We fit the law above on the smaller half of the $N$ grid, then predict loss and $B^*(N)$ on the larger half.
We also expect model capacity to modulate the recoverable term. A shallow tree cannot route through other columns to reconstruct a binned value, so for depth-limited trees we predict the gap behaves like the floor alone, even on high-redundancy data. If confirmed, this would explain why continuous handling matters most for interpretable models, which is the setting of ArborEnum and CLARITree.
Baselines
Naive reference: constant penalty. Binning costs a constant amount of loss independent of $N$. We fit this as a single offset per $(B, \text{dataset})$ on the small-$N$ half and extrapolate. This is the "do nothing" model of the interaction. If our law does not beat it on held-out large-$N$ error, the redundancy story adds nothing.
Separable scaling law. Loss is the sum of a power law in $N$ and a power law in $B$ with no interaction. This is the natural transfer of the form used in Scaling Laws for Precision and of the parameter-and-data law that TabDPT reports for tabular foundation models. It is the strongest structural competitor because it is what a practitioner would fit by default. It gets the same data, the same small-$N$ fitting split, and the same number of free parameters plus one. Beating it means lower mean absolute extrapolation error on held-out large-$N$ log-loss or RMSE across datasets, and a correctly signed correlation between fitted floors and $\sum_j w_j(1-R^2_j)$.
Classical univariate bin rule. This predicts $B^*(N)$ growing as a fixed root of $N$, as in histogram density estimation. It ignores the other columns. We beat it if our predicted $B^*$ is closer to the empirical best bin count on held-out sizes.
Strongest predictive system. Gradient-boosted trees remain the benchmark to clear. On OpenML-CTR23, OpenML-CTR23 – A curated tabular regression benchmarking suite reports an average RMSE rank of 1.31 for XGBoost, against 2.46 for random forests. We run XGBoost with max_bin swept and tuned under the same per-cell budget as every other model, with a fixed number of random-search trials. The continuous-feature references are:
- MLP with periodic embeddings, whose average rank improved from 8.5 to 3.0 in On Embeddings for Numerical Features in Tabular Deep Learning.
- TabDPT, which reports average zero-shot AUC 0.933 on OpenML-CC18 versus 0.926 for tuned XGBoost.
We are not trying to beat these on accuracy. They define the continuous ceiling against which $\Delta$ is measured.
Key ablation. We remove the redundancy term by setting all $R^2_j = 0$, and separately replace measured $R^2_j$ with shuffled values from other datasets. If predictions do not degrade, redundancy is not the mechanism.
Why not just…?
Why not just use continuous embeddings and stop discretizing? For many models we can, and periodic embeddings work well per On Embeddings for Numerical Features in Tabular Deep Learning. But discretization is often imposed rather than chosen:
- Rule lists such as Falling Rule Lists need binary conditions to stay readable.
- Token-based tabular models need a vocabulary.
- Text serialization in TabLLM is itself a discretization, and that paper reports tree models overtaking it at 256 or more labeled examples.
- Coarsened clinical cutoffs are frequently all that is recorded.
The question in all these cases is how much is lost and whether scale helps. Continuous embeddings do not answer that.
Why not just use more bins? For flexible models this is a real threat. If XGBoost with max_bin of 256 already sits at the continuous ceiling at every $N$ we test, the practical message collapses to "use enough bins," and we will report it as such. Our law still predicts how many are enough as $N$ grows, which the constant-penalty and univariate baselines do not. For exhaustive interpretable search, finer grids are not free: ArborEnum notes that depth-4 trees over 20 binary features already span $8.4\times10^{18}$ candidates.
Why not just rely on classical nonparametric theory? Single-feature bias-variance arguments give a bin-count rate. They treat the lost within-bin information as gone, which predicts a floor on every dataset. The redundancy term is exactly where we expect classical theory to be wrong on correlated real tables. The univariate-rule baseline and a synthetic control, described below, settle this directly.
Why not just plot learning curves per dataset? Curves describe behavior but do not transfer. The value of the law is predicting the large-$N$ floor and $B^*$ from small-$N$ runs plus a pre-computed redundancy score. The extrapolation test measures exactly that.
Proposed Experiments
Datasets. The primary large-$N$ testbeds are the Gorishniy suite datasets with enough rows to span three decades, including Covertype, Adult and California Housing; the 24-million-row synthetic credit card transactions of Tabular Transformers for Modeling Multivariate Time Series; and Beijing PM2.5 for regression. OpenML-CTR23 and OpenML-CC18 provide breadth across smaller tables. CTR23 datasets cap at 100,000 rows, so they contribute mostly to the cross-dataset floor-versus-redundancy correlation rather than to long curves.
Metrics. Test log-loss or RMSE, the gap $\Delta(N,B)$ to the continuous version of the same model, extrapolation error of each fitted law, and absolute error in predicted $B^*$.
First decisive check: synthetic control. We generate data with a known smooth target and directly vary the correlation between the binned feature and an unbinned proxy. Our law predicts the large-$N$ gap falls monotonically with the proxy's $R^2$ and vanishes at $R^2=1$ for flexible models. The classical view predicts no dependence. This experiment cleanly separates the two.
Second decisive check: real data, before fitting. We compute redundancy scores on all real datasets before any scaling run, then test the rank correlation between the score $\sum_j w_j(1-R^2_j)$ and the fitted asymptotic gap.
Third decisive check: capacity. We repeat the analysis with depth-3 trees, binarized versus CLARITree-style continuous splits. We predict a floor that does not shrink with $N$ even on high-redundancy datasets.
Falsification. The proposal fails if the separable law matches ours on extrapolation, or if measured redundancy does not predict which datasets plateau. Either outcome would be informative: it would mean the "data fixes binning" belief and its opposite are both governed by something other than redundancy among features.
References
- Regression-based Deep-Learning predicts molecular biomarkers from pathology slides
- Creatinine clearance versus serum creatinine as a risk factor in cardiac surgery
- Characterization of Non-Monotonic Relationships between Tumor Mutational Burden and Clinical Outcomes
- Scaling to Very Very Large Corpora for Natural Language Disambiguation
- Generating QM1B with PySCFIPU
- Scaling Laws for Precision
- On Embeddings for Numerical Features in Tabular Deep Learning
- Tabular Transformers for Modeling Multivariate Time Series
- CLARITree: Cholesky and Lookahead Accelerations for Regression with Interpretable Piecewise Linear Trees
- ArborEnum: Decision Tree Rashomon Sets over Continuous Features
- TabDPT: Scaling Tabular Foundation Models on Real Data
- OpenML-CTR23 – A curated tabular regression benchmarking suite
- Falling Rule Lists
- TabLLM: Few-shot Classification of Tabular Data with Large Language Models