Does the Layout Matter? Testing Tabular-to-Image Models When Expert Labels Are Scarce

A small but persistent line of clinical machine learning turns rows of a spreadsheet, such as a patient's cognitive test scores or examination findings, into small images, then trains an image classifier on them. The stated reason is that convolutional networks, which look at neighbouring pixels together, can pick up interactions between related features if related features are placed next to each other. Several single-cohort papers report that these "tabular images" beat conventional models. No study has checked the mechanism they rely on. If the gain comes from the layout, then a network trained on the same pixels in a random order should do worse. None of the published comparisons runs that control, and none tests on an external hospital.

We propose a controlled, preregistered study built around that control. It targets the regime where the approach would matter most: diagnostic labels need expert consensus and are therefore scarce (tens to a few hundred labeled patients), while unlabeled records are plentiful. The constructive half of the proposal follows from one observation. The layout depends only on correlations between features, and those correlations can be estimated from unlabeled records or from a different hospital. That makes the layout a candidate way to inject structure learned without any expert labels. We will test whether this label-free structure helps a small-sample classifier and whether it survives a change of site. If it does not, the field gets a clear negative result that it currently lacks.

Core Claim

The claim to be tested is specific. A correlation-preserving layout gives a measurable advantage over a label-matched, compute-matched random-permutation layout. That advantage grows as the number of expert labels shrinks, and it persists under leave-one-site-out evaluation.

We summarise this with a single quantity, the layout gain $\Delta(n)$. It is the paired difference in held-out AUC between the structured-layout network and the permuted-layout network when both are trained on the same $n$ labeled patients. The structured layout must also at least match a tuned gradient-boosted tree ensemble at $n \le 100$. If $\Delta(n)$ is indistinguishable from zero, the inductive-bias story behind this literature is unsupported, whatever the headline accuracy.

What Prior Work Establishes, and What It Leaves Open

The motivating papers each report a win on a single cohort, with no permutation control.

  • The Map-Optimize-Learn pipeline (Serra Neto et al.) orders pediatric cardiology features by distance and searches for a grid shape. Its CNN reached 0.925 test accuracy, against 0.904 for a plain neural network and 0.872 for logistic regression. The grid it selected was $6 \times 1$, effectively a one-dimensional strip, and it was never compared against boosted trees.
  • Choi et al. place neuropsychological scores along a Hilbert curve, a path that visits every cell of a grid while keeping consecutive items close. Their 2D-CNN reached an AUC of 0.9863 on KLOSCAD and outperformed XGBoost, random forest and SVM. The evaluation used one cohort and no external site.
  • IGTD (Zhu et al.) supplies the layout method we reuse. It assigns features to pixels so that pixel distances track feature correlations.
  • SuperTML (Sun et al.) renders feature values as text on a canvas so that pretrained vision models can read them. It supplies a second arm, one that relies on pretraining rather than layout.

Several papers constrain the design.

  • The tabular deep learning survey (Borisov et al.) and MulTaBench both treat gradient-boosted trees as the standard to beat on tabular data. Any claim that ignores them is weak.
  • Tartaglione et al. showed that at $N=447$ deep models learned which hospital an image came from rather than the disease, and that ResNet-50 underperformed ResNet-18. This is why we insist on held-out sites and small networks.
  • A Systematic Review of Deep Learning in Medical Imaging documents the absence of head-to-head comparisons. The tabular-image literature inherits this problem.

The reproducibility infrastructure we adopt comes from two toolkits.

  • MEDS-DEV declares tasks in a dataset-independent form and evaluates them across separate clinical databases.
  • PyHealth provides a fixed pipeline running from data loading through to evaluation.

We deliberately do not fuse the tabular images with scans. HyperFusion and TGV both show that tabular data can help imaging models. They were validated on 2,120 subjects and 49,737 scans respectively, far outside the $N<100$ regime. Stacking a rendered tabular tile as an extra channel on a scan would also carry no real spatial information, because pixel position in the tile has nothing to do with pixel position in the anatomy. A skeptic would rightly note that feeding the raw feature vector into the network would do the same job more simply. We therefore restrict the question to where the layout hypothesis can actually be falsified.

Research Program

The program has four parts.

Benchmark. We will build a small-sample benchmark from public, multi-site tabular cohorts in the file where expert-assigned labels are the costly resource.

  • ADNI. We use the cognitive-test, CSF-biomarker and demographic tables with clinician-assigned diagnoses of cognitively normal, mild cognitive impairment (MCI) or Alzheimer's disease, across the study's acquisition sites.
  • MAGGIC. Its 30 heart-failure studies give a natural leave-one-study-out structure, following Bellot and van der Schaar.
  • MIMIC-IV and eICU. Tasks are declared through MEDS-DEV predicates so that other groups can rerun them on their own databases.
  • KLOSCAD and the Brazilian pediatric dataset. Both are included only if access permits, so that we can attempt a direct replication of Choi et al. and Serra Neto et al..

Label sweep. For every task we draw 20 random labeled subsets at each of $n \in \{25, 50, 100, 200, \text{all}\}$. All remaining training-site records are available as unlabeled data.

Layouts. We estimate layouts in three ways, which separates the source of the structure from its use:

  • from the labeled subset only;
  • from all unlabeled training-site records;
  • from a different site entirely.

Each layout is rendered with IGTD and with a correlation-ordered Hilbert curve. The same small CNN, a ResNet-18-scale network or smaller, is trained on each layout. Because none of the estimation routes uses labels beyond the subset, the unlabeled and cross-site routes measure whether structure obtained without expert labels transfers.

Controls. Each structured layout is paired with two controls:

  • a random-permutation layout, repeated over 10 seeds, with identical pixels, network, schedule and tuning budget;
  • the structured layout with every convolution replaced by a $1\times1$ convolution. This keeps the pixels but removes the network's ability to combine neighbouring cells, which isolates spatial locality itself.

Baselines

Naive reference. L2-regularised logistic regression on standardised features with median imputation, tuned by nested cross-validation. It is the model a clinician-statistician would fit first. Beating it is necessary but proves little.

Strongest competitor. A tuned gradient-boosted tree ensemble (CatBoost), which both Borisov et al. and MulTaBench identify as the standard on tabular data. For the small-site survival setting the strongest competitor is Transfer Survival Boosting. It reports a C-index of 0.677 on the MAGGIC "Euro" cohort (under 200 patients), against 0.500 for a target-only Cox model (Bellot and van der Schaar). For tabular-only ADNI diagnosis, and for tuned boosted trees at $n \le 100$ on any of these cohorts, we found no reported result on Lacuna. We will therefore establish those reference numbers ourselves. For KLOSCAD, the number to replicate is the 2D-CNN AUC of 0.9863 reported by Choi et al..

Fairness conditions. Every method gets the same labeled subsets and the same 50-trial hyperparameter search, inside nested cross-validation. Every method also gets the same compute cap, preprocessing and imputation code, all run through a single PyHealth-style pipeline. "Beating" CatBoost means a paired AUC difference whose 95% bootstrap interval excludes zero, pooled over the 20 subsets. This must hold at $n \le 100$ on a majority of held-out sites.

Strong neural competitor. A multilayer perceptron given the same access to unlabeled records, pretrained by masking and reconstructing random features. If the images help only because they let us use unlabeled data, this network should match them.

Pretrained-vision competitor. A SuperTML-style rendering fed to an ImageNet-pretrained network, as in Sun et al.. It tests whether pretraining rather than layout drives any gain.

Key ablations. The permutation and $1\times1$ controls described above. They are what the core claim stands or falls on.

Why not just…?

Why not just use tuned gradient-boosted trees and skip the images? This is the real threat, and we treat it as such. The only reported wins of tabular images over boosted trees in the file come from a single cohort (Choi et al.). The other headline result (Serra Neto et al.) never compared against trees at all. We expect CatBoost to win at moderate and large $n$. The open question is whether label-free layout structure helps below roughly 100 labels, where trees have little data to split on. The label sweep answers this directly. If CatBoost wins at every $n$, we report that as the main finding.

Why not just pretrain an MLP on the unlabeled records instead of drawing pictures? If the benefit of our unlabeled-layout arm comes from using unlabeled data rather than from spatial arrangement, masked-feature pretraining of a plain network should capture it just as well. No paper in the file runs this comparison. We include the pretrained network with an identical unlabeled pool. If it matches the structured CNN while the permuted CNN lags, the correct conclusion is that the unlabeled data matters and the image does not.

Why not just fuse the tabular data with the scans, as HyperFusion does? HyperFusion reports balanced accuracy of 0.673 against 0.579 for images alone on ADNI. It generates network weights from tabular inputs, which adds many parameters. It was trained on thousands of subjects, the very setting where Tartaglione et al. show that extra capacity does not rescue small cohorts. Fusion is a different question from ours. We exclude it because it cannot test the layout hypothesis, not because it is inferior.

Proposed Experiments and Decisive Checks

The first decisive check is the permutation null. For each task, layout method and $n$, we compute $\Delta(n)$ with paired bootstrap intervals across the 20 labeled subsets and 10 permutation seeds. The hypothesis survives only if $\Delta(n)$ is positive and increases as $n$ falls. The $1\times1$ control should reproduce the permuted result: once the network cannot combine neighbouring cells, the layout should stop mattering. If it does not, any gain comes from something other than spatial locality.

The second check is transfer. We compare layouts estimated from unlabeled same-site records, from labeled records only, and from a different site, all evaluated on held-out ADNI sites and MAGGIC studies. A layout that helps only when estimated at the test site is a site shortcut, not a structural prior.

The third check is replication. On KLOSCAD, if accessible, we rerun the published Hilbert-curve pipeline at full $N$ and at our label budgets, adding both controls and CatBoost. The question is whether the reported 0.9863 AUC advantage over tree baselines survives matched tuning.

Everything will be released as MEDS-DEV task configurations with fixed split seeds, so that other groups can add their own databases as new held-out sites. The expected contributions are twofold. The first is a direct answer to whether the picture in tabular-to-image methods does any work. The second is a reusable small-sample, label-scarce benchmark on which future claims of this kind must survive a random-layout control.