Models that read relational databases directly, called relational foundation models, turn each entity (a customer, a product) and its linked rows into a sequence a Transformer can process. To fit in memory they keep only a fixed number of linked rows per entity, the "neighborhood budget." For classification this is usually harmless. For predicting a continuous number, such as next month's income from a transaction history, it can be fatal: the very customers with the most activity get cut to a small, arbitrary sample, and the quantity that determines the answer (a total, a rate, a trend) is thrown away before the model sees it. We propose to give these models, alongside their sampled raw rows, a handful of cheap "sketch tokens." These are streaming summaries (counts, sums, moments, quantiles, recency-bucketed totals) computed over each entity's entire neighborhood, whatever its size. The question is whether this recovers what human SQL aggregation recovers without anyone having to guess the target's functional form in advance.
Measuring this honestly needs a benchmark whose answers are known and cannot have leaked into anyone's pretraining data. Existing contamination-proof evaluations generate fresh problems on demand, but only discrete ones: arithmetic, logic, graph reachability. We therefore build a generator of synthetic relational databases with continuous targets. It controls two things: how skewed entity activity is, and which statistic of the history actually drives the target. Every evaluation gets fresh seeds and, optionally, never-seen schemas. The benchmark is the evaluation lens. Sketch tokens are the method. The generalist-versus-specialist comparison is the question both are used to answer.
Core Claim
The collapse of relational foundation models on regression is mostly not a property of regression. It is a mismatch between what the budget-limited sample contains and the summary statistic the target depends on. The evidence that motivates this is Context Window Failures in Relational Foundation Models. On the synthetic Animus financial database, the Relational Transformer, Griffin and RelGT all reach $R^2 \le 0.18$ on raw data when predicting continuous credit income. RelGT runs out of memory. Manual monthly SQL aggregation lifts the Relational Transformer to 0.65. The same paper shows that for customers with fewer than 16 transactions, raw events beat aggregates, because aggregation erases fine-grained temporal variation.
Taken together, these results say neither representation alone is right. What is needed is raw rows where they fit, and summaries over everything else. Our claim is that raw-rows-plus-sketches matches manual aggregation on high-activity entities, matches raw input on low-activity entities, and degrades gracefully on target families where any fixed aggregate is wrong. The last condition is what separates it from automated feature engineering.
Two lines of work constrain the design. RelBench introduced message-passing graph networks over databases and reports that they struggle more with regression than classification. It also reports that manual feature engineering costs an expert about 12.3 hours and about 878 lines of code per task. That cost is why "just aggregate by hand" is not an acceptable answer. IM-Context shows a U-shaped error curve in in-context regression: past an optimum, adding context examples raises error on rare target values. So "more rows is better" cannot be assumed, and the policy for choosing which raw rows fill the budget is itself a variable we test.
Research Program
The generator. The generator is modeled on Animus, which tracks 100,000 customers over 12 months. There, 60% of users have one transaction a month and 5% have 1,000 to 10,000. We keep that heavy-tailed activity profile as a tunable skew parameter and add multi-table schemas: customers, accounts, merchants, and two-hop links.
The key addition is a family of target functions, borrowing the idea of dynamically generated tasks with controllable complexity from DyVal. The families are:
- additive targets: the target depends on a sum of transaction amounts;
- rate targets: a count per time window;
- trend targets: the slope of recent activity;
- rare-event targets: driven by a few extreme transactions that uniform sampling tends to miss;
- order-dependent targets: the sequence of event types matters;
- cross-hop targets: the target depends on merchants' histories, not the customer's own.
Noise level and label imbalance are also controlled, which lets us report tail error in the style of the imbalanced-regression benchmarks used by IM-Context.
Contamination control. Every reported number uses fresh random seeds. A second split mirrors the table and column semantics of public RelBench schemas, and a third uses invented schemas. This lets us test whether pretrained relational models do better on familiar-looking schemas. Such an effect would be a contamination-like artifact of pretraining on public databases, and it relates to the ranking-reliability concerns raised for RelBench and DBInfer in Are Tabular Foundation Model Rankings Reliable?.
Sketch tokens. For each entity and each edge type, a single streaming pass over its full neighborhood computes:
- counts;
- sums;
- the first two moments of each numeric column;
- a small quantile summary;
- the same statistics restricted to a few recency buckets.
Each summary is encoded as an extra token with a type embedding, next to the budget-limited raw rows. Cost grows linearly in the number of rows and memory is constant per entity, which avoids the materialization that makes RelGT run out of memory.
The raw-row budget is filled by one of three policies: uniform sampling, most-recent-first, or inverse-density sampling over row values (adapted from IM-Context). We implement sketch tokens inside the Relational Transformer and, as a second host, in a GraphSAGE model, where sketches are node features.
The generalist-versus-specialist axis. Each host is run in two ways: pretrained and used with minimal fine-tuning, and trained from scratch on the target database. This tests whether the domain-specialized per-database model keeps its reported edge on raw data once both models see sketches.
Baselines
Naive reference points. The do-nothing baseline predicts each entity's own historical mean of the target, falling back to the population mean, plus a last-value predictor. Any learned model that cannot beat these on the high-activity bin is not reading the history at all.
Raw relational models. The Relational Transformer and Griffin run on raw rows under the default neighborhood budget, reproducing the Animus condition in which Context Window Failures in Relational Foundation Models reports $R^2 \le 0.18$. The same models are run with a budget matched in memory to our sketch-augmented variant, so any gain cannot be attributed to extra capacity.
Strongest reported method. The strongest reported result on this question is the Relational Transformer on manually aggregated Animus data, at $R^2 = 0.65$ (Agg-Simple, from the same paper). We re-run that pipeline on Animus. On generated databases we apply the analogous hand-written monthly aggregates. Beating it means reaching at least 0.65 on Animus with no hand-written SQL, while also beating Agg-Simple in the under-16-transaction bin, where that paper shows aggregation hurts.
Automated aggregation. A fixed library of per-edge-type aggregates, identical to the sketch statistics, feeds LightGBM on the flattened entity table. Here RelBench reports RDL beating single-table LightGBM by 20.4% on classification, but gives no regression number. No reported result on Lacuna was found for automated-aggregate LightGBM on Animus.
Specialized graph model. Per-database GraphSAGE from the RelBench pipeline is run at the largest budget that fits in memory. It serves as the specialized competitor that the Animus paper found beating the foundation models on raw data.
Fairness. All methods share data splits, the same GPU memory ceiling, and an equal hyperparameter search budget.
Ablations. Four ablations isolate the key ingredient: sketches with zero raw rows, raw rows with recency buckets removed from the sketches, sketches without type embeddings, and each of the three raw-row sampling policies.
Why not just…?
Why not just automate the aggregation and use LightGBM? This is the most serious threat. On additive and rate targets we expect it to tie with sketch tokens, since the features are the same. The proposal's value rests on the other families. Rare-event, order-dependent, and cross-hop targets are where a flat aggregate table loses information. Low-activity entities are where the Animus results show raw rows carry signal that aggregates erase. If automated aggregation with LightGBM matches sketch tokens on those families and bins too, we will report that the foundation-model route adds nothing for relational regression. That would be a useful negative result.
Why not just raise the budget, or use GraphSAGE with a big sample? Larger budgets help until memory runs out: RelGT did not finish on raw Animus. Uniform sampling of a 10,000-transaction history also still misses rare events. The matched-memory raw baseline and the budget sweep on the rare-event family directly measure how far budget alone goes.
Why not just evaluate on RelBench and Animus, without a generator? Static benchmarks cannot separate "the model learned the aggregate" from "the model has seen this schema." Animus has only one target form, so it cannot show which statistic a model fails to recover. DyVal found models with strong static scores dropping to near zero on freshly generated instances. The generator's schema split tests whether anything similar happens for pretrained relational models.
Why not just discretize the target and treat it as classification? Binning discards the within-bin ordering that regression metrics reward. It also does not address truncation: a model that sees 1% of a customer's transactions will still place a total in the wrong bin. Classification on binned targets is included as a secondary baseline, scored on the continuous error of the bin midpoints.
Proposed Experiments
Primary evaluation. The primary benchmark is Animus, together with 30 generated databases: five per target family, across three skew levels, split between RelBench-like and invented schemas. We report $R^2$ and mean absolute error, stratified by activity bin (the under-16 and over-16 split used in the Animus paper, plus a top-5% bin), and tail error by target frequency.
Decisive checks. Three results would settle the proposal:
- Sketch-augmented Relational Transformer reaches at least 0.65 $R^2$ on Animus without manual aggregation and does not lose to raw input in the under-16 bin.
- On rare-event and order-dependent families, it beats automated-aggregate LightGBM by a margin larger than the seed-to-seed spread.
- The gap between pretrained and scratch-trained models is similar on familiar and invented schemas. A significantly larger gap on familiar schemas would indicate schema memorization.
Secondary evaluation. RelBench entity-regression tasks provide real-data validation. There, success means improvement over the published RDL pipeline under the same compute. The RelBench paper reports no specific regression numbers for that pipeline, so we will establish the reference by re-running it.
References
- Context Window Failures in Relational Foundation Models
- RelBench: A Benchmark for Deep Learning on Relational Databases
- IM-Context: In-Context Learning for Imbalanced Regression Tasks
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Are Tabular Foundation Model Rankings Reliable? A Generalizability Theory Analysis of RelBench and DBInfer