Discovering Treatment-Effect Structure in Randomized Trials Without Data Dredging

When a randomized trial finds that an intervention works, the next question is almost always for whom and in what shape. Does the effect grow linearly with age, switch on above a threshold, or depend on two traits together? Today trialists answer this with hand-picked subgroup analyses, which invite the familiar problem of searching many subgroups until one looks significant. A compositional kernel grammar offers a principled alternative. This is a small algebra that builds a Gaussian-process prior (a probability distribution over smooth functions) by adding and multiplying simple pieces such as linear trends, smooth bumps and change-points, and then searches over these expressions. It could propose a readable formula for the treatment effect, such as "effect is constant, plus a step increase above age 21." The catch is that an open-ended structure search is the most aggressive form of data dredging ever automated. Its output means nothing inferentially unless the search itself is accounted for.

This note proposes to use the one thing a randomized trial gives that observational data do not: the assignment mechanism is known exactly. Because we know how treatment labels were drawn, we can redraw them under a null hypothesis of "no heterogeneity" and rerun the entire grammar search on each redraw. The spread of search outcomes under these redraws is a reference distribution for how much structure the search finds by chance. The result is a pipeline that proposes an interpretable treatment-effect formula and also attaches a p-value that already pays for having searched. We think this is the version of the grammar-meets-trials idea worth doing. Simply plugging a grammar into an existing causal Gaussian-process model would improve fit, but it would make the inference problem worse.

Core Claim

Grammar search over the kernel of the treatment-effect function, embedded in a Gaussian-process partially linear model and coupled to a randomization test that reruns the full search on re-drawn treatment labels, yields two things. First, it gives a test of effect heterogeneity with nominal Type I error at trial sizes of 100 to 500. Second, when heterogeneity is real and structured (thresholds, single-covariate trends, two-way interactions), this test has more power than splitting the sample into a search half and a test half. It also recovers the correct structural form, such as additive versus interactive, more often than fixed-kernel or black-box estimators. We do not claim large gains in pointwise effect accuracy over strong existing estimators. The contribution is calibrated, readable structure.

Research Program

The outcome model follows Uncertainty Quantification in Heterogeneous Treatment Effect Estimation with Gaussian-Process-Based Partially Linear Model. That work writes the outcome as a treatment effect function $\theta(x)$ times treatment, plus a nuisance baseline $f(x)$, plus noise. It places Gaussian-process priors on both, integrates out $f$ analytically, and reports calibrated intervals at $n=200$, but it uses fixed, hand-chosen kernels. We keep its closed-form posterior and change two things. First, because the trial's assignment probability $e$ is known, we code treatment as $T-e$ rather than $T$. This makes the treatment regressor uncorrelated with every function of the covariates by design, so a misspecified baseline $f$ inflates variance but cannot leak into $\theta$. Observational applications of that model cannot use this device. Second, the kernel on $\theta$ is chosen by a grammar search in the style of Structure Discovery in Nonparametric Regression through Compositional Kernel Search and The Automatic Statistician. Those works supply the search procedure: greedy expansion of kernel expressions, marginal-likelihood fitting of hyperparameters, and a Bayesian information criterion penalty. Neither addresses causal targets or selection-aware inference.

For tabular trial covariates, the grammar's building blocks act on one covariate at a time. They are a constant (a homogeneous effect), a linear kernel, a squared-exponential kernel (smooth local variation) and a change-point operator (a threshold). These combine by addition and multiplication up to depth three, with products restricted to at most two covariates so that every expression reads as a sentence. The baseline $f$ gets a fixed flexible kernel: a squared-exponential kernel with a separate length scale per covariate, plus a linear kernel. The reason is that $f$ is a variance-reduction device, not the object of interest. At $n\le 500$, exact inference is cheap. A search visiting roughly 50 to 100 candidate expressions takes seconds, which is what makes rerunning it hundreds of times feasible. The scaling problems catalogued in Scaling up the Automatic Statistician do not bind in this regime.

The test statistic $S$ is the gain in log marginal likelihood of the best expression the search finds over the constant-effect model:

$$S = \max_{k \in \mathcal{G}} \log p(Y \mid k) - \log p(Y \mid k_{\text{const}}),$$

where $\mathcal{G}$ is the set of expressions the greedy search visits and $k_{\text{const}}$ is the constant kernel on $\theta$. The null hypothesis is that the treatment effect equals the same unknown constant $\tau$ for every participant. For a candidate $\tau$, we impute both potential outcomes for everyone, $Y(0)=Y-\tau T$ and $Y(1)=Y(0)+\tau$. We then redraw $T$ from the actual design, preserving any stratification or blocking, recompute observed outcomes and rerun the full search to get $S^{(b)}$. The p-value is the fraction of redraws with $S^{(b)}\ge S$. Because $\tau$ is unknown, we take the maximum p-value over a 99.9% confidence interval for $\tau$ and add the interval's 0.1% miss rate. This standard confidence-interval correction keeps the test valid for any finite sample rather than only asymptotically. A cheaper plug-in version using $\hat\tau$ alone is also reported, to measure how much the correction costs.

For interpretation, the winning expression is rendered as text with component curves, as in the Automatic Statistician. We also report how often each component reappears when the search is rerun on bootstrap resamples. This separates a stable claim like "threshold on age" from a fragile second-order term.

Baselines

The naive reference point is the analysis most trials actually report: a difference in means with a set of pre-specified subgroup interaction tests, corrected by Bonferroni. For heterogeneity detection it is the honest floor. It is valid, powerless against structure it did not anticipate, and silent about shape. It will be given the same covariates, with subgroups defined by median splits, and scored on Type I error and power.

The strongest competitor for effect estimation is the fixed-kernel Gaussian-process partially linear model of Horii and Chikahara. We run it with a per-covariate-length-scale squared-exponential kernel on $\theta$, on the same data, with the same treatment coding and hyperparameter optimizer and the same number of optimizer restarts per fit. It isolates exactly what the grammar adds. The file reports it outperforming Bayesian Causal Forests and the R-Learner in mean squared error and interval coverage at $n=200$, notably on the non-differentiable Setup D, but gives no numeric table. So for these setups there is no reported result found on Lacuna, and we will reproduce it ourselves. For its heterogeneity test we apply the same full randomization procedure with its marginal likelihood gain as the statistic. This fixed-kernel randomization test is the key ablation: any power difference from our method is attributable to the grammar, not to the randomization machinery.

For pointwise accuracy on IHDP-style data, the strongest numeric reference in the file is HyperTLearner from Dynamic Inter-treatment Information Sharing for Individualized Treatment Effects Estimation. It uses a hypernetwork to share parameters across treatment arms and reports an out-of-sample error in estimated individual effects (PEHE-out) of 1.12 on IHDP, against 1.32 for a plain T-Learner. We also run CausalPFN, a transformer pretrained on simulated causal datasets that estimates effects in one forward pass. It reports the best average PEHE rank across 310 tasks, but the file gives no scalar. We use both with their released defaults and report PEHE. Matching roughly 1.12 on standard IHDP while additionally providing a calibrated structure test would count as success. We do not require beating it. The second ablation is sample splitting: grammar search on half the trial, then a single pre-committed test of the chosen expression on the other half. Beating it on power at equal nominal level is the central claim.

Why not just…?

Why not just split the sample, search on one half and confirm on the other? This is valid and simple, and it is the most serious threat. At $n\approx 100$–$200$, typical of the trials in the file such as the 117-person exergame study (Effects of Playing Exergames on Quality of Life among Young Adults) and the 90-child ADHD trial (Empowering Social Growth Through Virtual Reality–Based Intervention for Children With ADHD), halving the data both degrades the search and halves the test's sample. The full-sample randomization test uses every participant for both jobs. If splitting matches our power across the simulation grid, the method's inferential rationale collapses and we would report the grammar only as a descriptive tool.

Why not just use a fixed flexible kernel and read the length scales? Separate length scales per covariate on $\theta$ do indicate which covariates matter. But they cannot distinguish a threshold from a trend, or an additive effect from an interaction. That distinction is precisely what Really Doing Great at Estimating CATE? shows drives which estimators win: on IHDP, making the effect additive flips the ranking between neural and forest methods. The fixed-kernel randomization-test ablation measures directly whether structure search buys power. Structure recovery rate measures whether it buys correct shape.

Why not just use a causal forest or CausalPFN for heterogeneity? Both estimate effects well, but neither produces a formula. The file notes CausalPFN is a black box restricted to binary treatment. Forests, per Horii and Chikahara, have weaker small-sample uncertainty. Either could be wrapped in the same full randomization test, and we will do this for a causal forest using a variance-of-predicted-effects statistic. If it matches our power, the grammar's remaining value is interpretability alone, and we will say so.

Why not just fit a mixed model with pre-specified interactions? For the question of "which shape," pre-specification is the gold standard only when the shape is known in advance. The naive baseline above covers this case and will show where it loses.

Proposed Experiments

The simulation study uses the Nie and Wager Setups A–D at $n=200$ and $500$, as used by Horii and Chikahara. We add a family of randomized trials built on real IHDP covariates. In these, we assign treatment by a known Bernoulli or stratified design, and set the true effect to be one of four cases: constant, a single-covariate threshold, a linear trend, or a two-covariate product. Each case appears in additive and non-separable baseline versions, mirroring the manipulation in Really Doing Great at Estimating CATE?. Across 500 replicates per cell we measure four things. The first is Type I error under the constant-effect cases at nominal 5%. The second is power under the others. The third is whether the top expression matches the true structure, meaning the same covariates and the same additive or product form. The fourth is PEHE and 90% interval coverage. On the original IHDP benchmark we report PEHE against HyperTLearner's 1.12, but not the randomization test, because the assignment mechanism there is not known.

The decisive checks are three. If the search-aware test exceeds 6.5% rejection under the null in any cell with $n\ge 200$ (well outside Monte Carlo error for 500 replicates), the plug-in or confidence-interval construction is broken. If sample splitting or the fixed-kernel randomization test matches grammar power within Monte Carlo error across structured cells, the proposal's core claim fails. If structure recovery is not clearly above the fixed-kernel method's ability to pick the right covariate, the interpretability claim fails.

A real-data case study is contingent on data access. We would ask the exergame authors for their data, since they report subgroup findings such as general-health change of 8.81 versus 0.00 in low-enthusiasm participants at $p=0.02$, and ask whether a search that pays for its own flexibility still supports any heterogeneity claim.