Honest Brushing: Valid Subgroup Mediation Effects from Interactive UMAP Views

Researchers who study why media changes behavior keep finding that exposure alone does nothing. A travel video raises travel intention only when it makes viewers mentally rehearse the trip. A film draws tourists only when viewers empathize with its characters. The middle step, called the mediator, does the work. The existing studies estimate one average mediated effect per sample. Yet their own results show that the effect flips across subgroups: slow music increases mental rehearsal for narrative activity videos and decreases it for non-narrative landscape videos (Synergistic Effects of Content Structure and Sensory Elements). The obvious next tool is an interactive dashboard. It would project each participant's high-dimensional response into 2D with UMAP (a nonlinear method that places similar points near each other), let analysts select clusters with the mouse ("brushing"), and update linked panels showing the mediated effect for the selected group. Such a tool invites false discoveries. The analyst chooses a region after looking at the data, UMAP can split continuous structure into apparent clusters, and the effect is then estimated on the same points that shaped both the picture and the choice.

This proposal builds and tests a fix: an inference layer under brushing, so that every region an analyst selects in a UMAP view reports a mediated effect with a valid confidence interval and a stability score. The central question is empirical. Do brushed subgroup effects from naive linked-view exploration fail to replicate at high rates? Does separating the data that draws the map from the data that estimates the effect, combined with checking how stable the layout is, restore valid error rates without destroying the analyst's ability to find real heterogeneity?

Core Claim

Coordinated multiple views (several linked charts sharing one selection state) are the main way analysts interact with UMAP projections. When the quantity shown in the linked panel is a causal mediation effect, brushing is a form of post-hoc subgroup selection. We claim that two ingredients make the displayed effect trustworthy. The first is a strict split between data that trains the representation and layout and data that is projected onto the fixed layout and used for estimation. The second is a per-selection stability score computed across UMAP seeds and neighborhood settings. We expect the in-sample estimate shown by a naive dashboard to have substantially worse interval coverage on planted ground truth. We also expect the honest version to keep most of the power to recover real regions.

The novelty is not "a dashboard for mediation." It is a measured answer to the problem the research file calls Gap 3: whether a brushed selection in a distorted 2D projection corresponds to real mediation heterogeneity or to an artifact of the projection.

Research Program

Representation stage. We take an existing deep mediation model as the representation learner. DMAVAE splits observed covariates into separate hidden confounder blocks for the treatment–mediator, treatment–outcome and mediator–outcome links. This relaxes the "no hidden confounding" assumption, called sequential ignorability, that classical structural equation models rely on. IMAVAE handles a mediator that is itself high-dimensional and seen only through proxies. Neither model offers any way to inspect its latent space, and we reuse them rather than compete with them.

Layout stage. UMAP is fit on fold A latents only. Fold B points are placed onto the frozen layout with UMAP's out-of-sample transform. Choosing UMAP is a constraint rather than a preference. Unexplainable Explanations shows that UMAP behaves as a fast approximation to a double-kernel locally linear embedding. It also shows UMAP is highly sensitive to noise and to its neighborhood and spacing parameters (n_neighbors, min_dist), and can tear continuous trajectories into separate clusters. That sensitivity is the failure our stability score must detect.

Estimation for a brushed region. For a region $R$, the dashboard reports

$$\widehat{\mathrm{NIE}}(R)=\frac{1}{|B_R|}\sum_{i\in B_R}\left[\hat\mu(1,\hat m_i(1),x_i)-\hat\mu(1,\hat m_i(0),x_i)\right].$$

Here $B_R$ is the set of fold B points inside $R$. $\hat m_i(t)$ is the predicted mediator under treatment $t$. $\hat\mu(t,m,x)$ is the predicted outcome given treatment $t$, mediator $m$ and covariates $x$. The nuisance models are cross-fitted within fold B, meaning each point's predictions come from models trained on other points. The interval comes from a bootstrap over fold B.

Stability score. The stability of $R$ is the mean Jaccard overlap between $B_R$ and the matching selection after re-running the layout over a grid of seeds and UMAP settings. A matching selection is found by mapping $R$'s fold A anchor points through each re-run. Selections below a pre-registered threshold are flagged in the interface.

Interface. The linked views are a stimulus pane (video keyframes, audio tempo, condition labels), the mediator map, a mediation pane (natural direct effect NDE and natural indirect effect NIE with honest intervals), and outcome histograms. Shared selection state follows State Synchronization for Multi View Data Exploration. The stimulus pane lets analysts check whether a region reflects a real stimulus pattern, such as narrative structure crossed with tempo, or only a layout artifact.

Baselines

Pre-specified analysis (do-nothing reference). This is the field's current practice: a whole-sample linear structural equation model, or a factorial ANOVA on pre-registered cells, as in the film-tourism empathy study and the offline-to-online clothing study. It cannot overfit to a selection, but it can only find heterogeneity someone specified in advance. Beating it means recovering planted regions it misses while matching its error rate. On the DMAVAE synthetic benchmark, linear SEM is reported at greater than 7% NDE bias, against about 1.5% for DMAVAE (Disentangled Representation for Causal Mediation Analysis). We reproduce those whole-sample figures as a sanity check.

Naive linked-view brushing. UMAP is fit on all data, and the NIE in the brushed region is estimated on the same points with the same deep mediation model. This is what the obvious dashboard would do, and it is the method we expect to fail. It runs with an identical encoder, identical estimators, identical compute and the same simulated brushing policy.

Strongest non-visual competitor. A cross-fitted subgroup-discovery tree is fit on per-unit indirect-effect estimates, with honest estimation of leaf effects. It is the most serious rival for finding valid heterogeneity at all. It receives the same folds and tuning budget as our method. No reported result was found on Lacuna for this baseline on these benchmarks, so we report our own runs.

Path-wise attribution. PWSHAP splits a prediction into direct and mediated path contributions, and it found a Race→Marital Status→Income effect of 0.361 on UCI Adult that standard Causal SHAP missed. We cluster its per-unit mediated-path attributions as a further heterogeneity baseline.

Ablations. Three ablations isolate the key ingredients: the data split without the stability score, the stability score without the split, and both together using PCA instead of UMAP.

What counts as winning. Interval coverage for brushed regions within 3 points of nominal 95% is the target. The naive dashboard should fall clearly below this. Region-recovery power should be at least that of the tree baseline on planted structures that are curved or non-axis-aligned.

Why not just…?

Why not just fit a subgroup tree on individual indirect effects and skip visualization? This is a real threat. On planted heterogeneity that lines up with the axes, it may match us, and if so we will say the inference layer's value lies in auditing exploration rather than in discovery. We expect it to lose where the heterogeneity follows curved structure in a high-dimensional mediator, as in the 616-dimensional IMAVAE electrophysiology data, which axis-aligned splits approximate poorly. It also cannot use what analysts see in the linked stimulus pane. The semi-synthetic experiments separate these two regimes explicitly.

Why not just use PCA, which is deterministic and needs no stability score? PCA removes the instability from seeds and settings but may merge curved subpopulations. The PCA ablation measures both effects directly: how many planted regions become recoverable, and whether the stability score was the only thing UMAP needed.

Why not just tell analysts to pre-register subgroups? Pre-registration gives up discovery, and the tempo reversal in Synergistic Effects is exactly the kind of interaction a whole-sample model hides. The pre-specified baseline shows the cost in missed regions.

Why not just put a multiple-comparison correction on in-sample brushing? The number of possible brushes is unbounded and depends on the data, so there is no fixed family of tests to correct over. Sample splitting avoids this, and the naive-plus-Bonferroni variant is included as a comparison.

Proposed Experiments

Experiment 1: semi-synthetic benchmarks with known answers. We plant heterogeneous indirect effects in three settings: curved manifold regions, axis-aligned regions, and a continuous effect gradient that UMAP may tear apart. The settings are the DMAVAE-style synthetic mediation benchmark (N = 10,000) with hidden confounding, UCI Adult with the outcome re-simulated, and the IMAVAE semi-synthetic electrophysiology data. A scripted brushing policy selects the regions that look most separated. It mimics an analyst and is matched across methods. Metrics are interval coverage of the true regional NIE, the false-discovery rate of "significant regional effect" claims, recovery power and stability.

Decisive check. The gradient setting is the decisive test. Naive brushing should report confident, opposite-signed effects for torn halves of what is really one gradient. Honest brushing should either cover the truth or flag the selection as unstable.

Experiment 2: a new factorial study. We replicate the structure × visual × tempo design of Synergistic Effects. We add per-participant free-text descriptions of what they imagined, embed them, and treat them as the high-dimensional mediator. Participants come from an age range wider than the 18–30 cohorts that limit the cited studies. The published tempo reversal serves as an external criterion that honest brushing should recover.

Experiment 3: analyst study. Analysts explore Experiment 2 data with either the naive dashboard or the honest one. We measure what fraction of their claimed subgroup effects replicate on a third, held-out fold.

Reference figures. No reported result was found on Lacuna for coverage or false discovery under interactive brushing. The figures we can anchor to are DMAVAE's about 1.5% NDE bias, IMAVAE's 5–25x error reduction over linear SEM, and UMAP's 0.829 k-NN accuracy on MNIST, used only as a reproduction check.