Neural networks for spreadsheet-style data improved sharply once each number was mapped into a richer vector before entering the network, instead of being fed in as a raw scalar. On Embeddings for Numerical Features in Tabular Deep Learning showed this with two encodings. The first is piecewise-linear encoding, a soft version of binning. The second is periodic encoding, which passes each value through learned sine and cosine functions. With either one, a plain multilayer perceptron moves from an average rank of 8.5 to 3.0 across eleven benchmarks and becomes competitive with gradient-boosted trees. Every one of those results is an average over the whole population. Nobody has asked who the gain goes to. Many tabular datasets contain subpopulations defined by a range of a numerical column, such as patients over 60 or houses in a rare price band. These are the subgroups that stratified analysis in medicine and economics cares about. A better encoding could help them disproportionately. It could also just lift everyone, or it could fit the majority more sharply at the minority's expense.
This note proposes to measure that directly, using the per-group scaling law of Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data as the measuring instrument. That law describes how each group's error falls as two things grow: the amount of that group's own data, and the amount of data overall. The question is whether the input encoding changes the shape of those curves, so that a minority group needs less of its own data because it borrows more from the rest of the population. It is not enough for the encoding to move the curves down uniformly. If the shape changes, the encoding becomes a lever on subgroup data requirements and not merely an accuracy boost. The question matters in practice, because over-representing minority groups in training data is the main known fix, and it is expensive.
Core Claim
Sagawa et al. model the risk of group $g$ as
$$R_g \approx \sigma_g^2 n_g^{-p} + \tau_g^2 n^{-q} + \delta_g,$$
where the symbols mean the following:
- $n_g$ is the number of training examples from group $g$, and $n$ is the total number of training examples.
- $\sigma_g$ and $p$ govern how much the group's error depends on its own data.
- $\tau_g$ and $q$ govern how much it improves from shared population data.
- $\delta_g$ is an irreducible floor.
That paper treats the model and its inputs as a black box that fixes these constants. Our claim is that the numerical encoding is one of the things that sets them.
There are two competing mechanisms, and they make opposite predictions.
The transfer hypothesis says that smooth periodic features let a minority group defined by a numerical range share structure with neighbouring values. This lowers the own-data term $\sigma_g^2 n_g^{-p}$ relative to the shared term $\tau_g^2 n^{-q}$. As a consequence, the training allocation that minimizes population or worst-group risk should over-represent the minority less.
The locality hypothesis says the opposite. Bin-like piecewise encodings give each value range its own parameters, which makes rare ranges more dependent on their own data and raises $\sigma_g$.
Which mechanism dominates, and whether it differs between piecewise-linear and periodic encodings, is the new empirical content of this proposal. A result in which the encoding shifts every group's curve down by the same amount, leaving the optimal allocation unchanged, would count as a negative result for both hypotheses. We would report it as such.
Relation to Prior Work
The closest work connects representation geometry to subgroup error, but only for images and text with discrete groups.
- Spurious Correlation-Aware Embedding Regularization for Worst-Group Robustness bounds worst-group error by how strongly the classifier aligns with spurious versus core directions in embedding space. It then regularizes training to reduce the spurious alignment. It requires group labels and binary spurious attributes. We do not regularize anything. We ask what an off-the-shelf input encoding already does.
- Representation Invariance and Allocation: When Subgroup Balance Matters shows that the separation between subgroups in the penultimate layer, measured by total-variation distance, predicts whether rebalancing helps. The reported Pearson $r$ is between 0.60 and 0.95 across MIMIC-CXR, HAM10000 and CivilComments. The authors explicitly restrict the theory to binary attributes and foundation-model embeddings. We reuse their diagnostic as a secondary measurement and test whether it carries over to subgroups defined by continuous moderators, where groups are quantile strata.
- Sagawa et al. 2021 supply the scaling-law method we reuse. Their datasets include Adult, which is also one of the benchmarks in Babenko et al. That overlap is what makes the comparison well posed.
Two papers constrain the design.
- Overparameterization hurts worst-group accuracy with spurious correlations shows that adding capacity can make worst-group error much worse under reweighting. On CelebA it degrades to above 60%, against 25.6% for a narrow model. Periodic encodings add parameters, so any effect we see must be separated from raw capacity.
- The Effect of Model Size on Worst-Group Generalization finds the opposite under plain training with pre-trained backbones. Our parameter-matched controls are designed so that we can say which regime input-side expansion falls into.
Research Program
We fix the backbone as an MLP and vary only the numerical encoding. There are five encoding conditions:
- raw scalars;
- raw scalars with a wider first layer, parameter-matched to the periodic encoding;
- piecewise-linear encoding;
- periodic encoding with linear and ReLU layers, as in Babenko et al.;
- periodic encoding with frozen random frequencies, which separates learned frequencies from the mere presence of sinusoids.
Each dataset receives two kinds of subgroup.
- Categorical groups: sex and race in Adult, following Sagawa et al.
- Continuous-moderator groups: quantile strata of a numerical column, such as the oldest age band in Adult, the highest housing-age or rare-location band in California Housing, and extreme elevation bands in Covertype. These are the cases where the encoding acts directly on the column that defines the group, so they are where an effect is most plausible.
For each encoding we run an allocation sweep. We vary the minority's share of the training set over a grid at several total sizes $n$. We train multiple seeds and evaluate on a fixed stratified test set. We then fit the scaling law per group and per encoding, with bootstrap confidence intervals on $\sigma_g$, $\tau_g$, $p$ and $q$. From the fitted curves we derive three quantities:
- the allocation that minimizes worst-group risk;
- the allocation that minimizes population risk;
- the "minority penalty", meaning the excess worst-group loss at the natural prevalence.
Alongside these fits we compute the total-variation separation between subgroups in the penultimate layer for each encoding. This tests whether separation explains the allocation sensitivity we observe, as Alloula et al. found for images and text.
Finally, a semi-synthetic variant probes the shortcut question from Sagawa et al.
- We inject a correlation between a numerical column and the label in the majority only, and ask whether richer encodings make the network latch onto that shortcut faster.
Baselines
Naive reference: raw-scalar MLP. This is the raw-scalar MLP trained by plain empirical risk minimization at the natural prevalence. Every claimed effect is measured as a change from this model's fitted scaling curves.
Strongest encoding baseline: MLP with periodic encoding at natural prevalence. This is the configuration from Babenko et al. Its reported advantage, average rank 3.0 against 8.5, is purely aggregate. No reported result was found on Lacuna for its worst-group or per-subgroup accuracy on any tabular benchmark. If we show only that it also improves worst-group accuracy, that is necessary but not sufficient. Beating it in the sense this proposal cares about means demonstrating a change in the fitted $\sigma_g/\tau_g$ ratio, or in the optimal allocation, with non-overlapping bootstrap intervals.
Strongest subgroup baseline: allocation-optimized training, compared against gradient-boosted trees. This is the intervention Sagawa et al. 2021 found more potent than reweighting or GroupDRO. We apply the best allocation found by the sweep to the raw-scalar MLP, and separately to CatBoost and XGBoost.
- Trees are the strongest tabular competitors in Babenko et al., and they bin numbers natively. This makes them a natural "locality" reference point.
- On Adult, the only shared dataset, no reported result was found on Lacuna for per-group accuracy, either from Sagawa et al. 2021 or from Babenko et al. All numbers for this baseline will therefore be produced by our own runs.
- Every method receives the same data splits, the same hyperparameter search budget (an equal number of trials), and early stopping on worst-group validation loss.
- The proposal's practical claim holds if an encoded MLP at natural prevalence matches the worst-group risk of an allocation-optimized raw MLP or tree model, while keeping more population accuracy.
Key ablation: parameter-matched wide raw-scalar MLP. This isolates whether any change comes from the encoding's structure or from extra capacity. A related ablation, the frozen-frequency periodic model, isolates whether the frequencies need to be learned. We add group reweighting and annotation-free last-layer retraining, following For Robust Worst-Group Accuracy, Ignore Group Annotations, on top of each encoding. This tests whether the encoding and the training-time fixes are complementary.
Why not just…?
Why not just use gradient-boosted trees, which already split on numerical ranges? This is a real threat. If CatBoost already has small minority penalties on continuous-moderator groups, the encoding question matters mostly to people committed to neural models. The allocation sweep runs trees through the same scaling-law fit, so the experiment answers the question directly. If trees dominate everywhere, we will report that trees and piecewise encodings share a locality advantage and that the periodic encoding adds nothing on subgroups.
Why not just over-represent the minority group in training? Sagawa et al. 2021 show this works. On Goodreads, the optimal allocation for the history group is about 30%, against a natural prevalence of 38%. The catch is that it requires collecting or reweighting group-specific data, which is often unavailable. Our question is whether the encoding reduces how much of that intervention is needed. Allocation is therefore treated as a baseline that the encoding is compared and combined with, not as a rival explanation. Sagawa et al. 2020 add that subsampling has a steep price: on CelebA it reached 15.1% worst-group error using only 3.4% of the data.
Why not just retrain the last layer on balanced or pseudo-labelled data? Stromberg et al. report about 83.5% worst-group accuracy on CelebA with RAD-UW, a last-layer method that needs no group labels. But last-layer retraining can only reuse features the backbone already learned. Combating Representation Learning Disparity with Geometric Harmonization shows that minority structure can collapse upstream, before any last layer sees it. We run last-layer retraining on top of every encoding. If it erases the differences between encodings, the encoding is irrelevant once retraining is applied, and we will say so.
Why not just conclude the encoding is better on average and stop? A uniform downward shift would raise worst-group accuracy without changing anything about subgroup data needs. This is why the primary outcome is the fitted curve shape and the optimal allocation, and not raw worst-group accuracy. We also compare encodings at checkpoints and model sizes where their average loss is matched.
Decisive Checks
The study succeeds or fails on four checks.
- Curve shape. For continuous-moderator groups, does any encoding change the fitted ratio of the own-data term to the shared term, with bootstrap intervals that exclude the raw-scalar value, after controlling for parameter count?
- Optimal allocation. Does the worst-group-optimal allocation move toward natural prevalence under periodic encoding (supporting transfer), or away from it under piecewise encoding (supporting locality)?
- Separation as a diagnostic. Does penultimate-layer subgroup separation rank the encodings by allocation sensitivity, extending Alloula et al.'s correlation to continuous strata?
- Shortcuts. In the injected-shortcut variant, does the encoded MLP rely on the shortcut more than the parameter-matched wide raw MLP? Greater reliance would place input expansion on the harmful side of the overparameterization debate.
References
- On Embeddings for Numerical Features in Tabular Deep Learning
- Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data
- Spurious Correlation-Aware Embedding Regularization for Worst-Group Robustness
- Representation Invariance and Allocation: When Subgroup Balance Matters
- Overparameterization hurts worst-group accuracy with spurious correlations
- The Effect of Model Size on Worst-Group Generalization
- For Robust Worst-Group Accuracy, Ignore Group Annotations
- Combating Representation Learning Disparity with Geometric Harmonization