Scale-Free Pre-Trend Equivalence Tests for Multi-Site Panels

Many policy and health evaluations compare outcome trajectories of treated and untreated groups before and after an intervention. This design is called difference-in-differences. It rests on a "parallel trends" assumption: without treatment, the two groups' outcomes would have moved in step. Analysts check this by looking at the years before treatment. When the panel pools several hospitals, cohorts or states, a further problem arises that the field has not addressed. Each site often records the outcome on its own scale. One hospital grades bleeding with one scale and another uses a different one. One registry codes "failure" after two cycles and another after three. Athey and Imbens showed that parallel trends depend on the scale chosen: the same data can pass or fail depending on whether one analyzes levels or logarithms (Identification and Inference in Nonlinear Difference-In-Differences Models). The consequence is that a pre-trend test run on pooled, unevenly measured outcomes partly measures differences in measurement protocol and only partly measures confounding.

We propose a pre-trend test whose verdict does not change when any site relabels its outcome through a strictly increasing transformation. The test works on ranks within each site. In each pre-treatment period it asks where treated units fall in a rank-based prediction built from the control group, then pools these rank-scale deviations across sites with a random-effects model that allows sites to differ. It then runs an equivalence test, which tries to show that the deviation is small rather than merely failing to show it is large. The two-sentence version is this. Express pre-trend violations in percentile units within each site, so that site-specific measurement scales cancel. Then demand positive evidence, pooled honestly across heterogeneous sites, that those violations lie inside a pre-stated percentile bound.

Core Claim

Three existing tools solve parts of the problem, and none combines them. Changes-in-changes (CIC), from Identification and Inference in Nonlinear Difference-In-Differences Models, models the outcome as $Y = h(U, T)$, with $h$ increasing in an unobserved unit-level "type" $U$. It predicts the treated group's counterfactual distribution by quantile mapping through the control group, and that prediction is invariant to monotone transformations of $Y$. CIC is an estimator for one population, though. It has no pre-trend test and no mechanism for sites whose measurement maps differ.

The equivalence testing of A Practical Guide to Counterfactual Estimators for Causal Inference with Time-Series Cross-Sectional Data uses two one-sided tests (TOST). TOST checks that a pre-trend falls inside a bound $\pm\delta$, which avoids the "failure to reject" trap of ordinary significance tests. But TOST there is stated on the additive outcome scale. A single $\delta$ in raw units is meaningless when sites use different units, and the method assumes measurement error behaves the same way across units.

The heterogeneity literature, finally, documents that inconsistent diagnostic and outcome definitions push researchers to abandon pooling altogether (The Challenge of Heterogeneity in Medical Data Synthesis). Harmonization tooling such as PyHealth: A Deep Learning Toolkit for Healthcare Applications maps codes onto common vocabularies. It cannot repair cases where two scales are different coarsenings of the same underlying severity.

Our claim concerns a statistic and its guarantee. For site $s$ and pre-treatment period $t$, let $\hat F^{\text{CIC}}_{s,t}$ be the CIC-predicted distribution of treated outcomes, built from treated period $t-1$ and control periods $t-1, t$ at site $s$. Define the rank-scale deviation as

$$\Delta_{s,t} = \mathbb{E}\left[\hat F^{\text{CIC}}_{s,t}(Y^{\text{treated}}_{s,t})\right] - \tfrac{1}{2}.$$

This is the average CIC-predicted percentile of the observed treated outcomes, minus one half. It is zero when the CIC identifying condition holds in that pre-period. It is unchanged if site $s$ replaces $Y$ with $g_s(Y)$ for any strictly increasing $g_s$, because ranks are preserved. The site-level deviations are then pooled with a random-effects model, $\Delta_{s,t} = \mu_t + \tau_s + \varepsilon_{s,t}$. Here $\mu_t$ is the common pre-trend violation in period $t$, $\tau_s$ is between-site heterogeneity in that violation, and $\varepsilon_{s,t}$ is sampling noise. The test declares equivalence only if TOST rejects $|\mu_t| \ge \delta$ for every pre-period and, separately, a prediction interval for a new site's $\mu_t + \tau_s$ lies inside a wider bound $\delta'$. The bound $\delta$ is stated in percentile points, for example "treated units drift by less than three percentiles per year relative to controls." That statement means the same thing at every site.

One subtlety needs to be stated directly. This procedure tests the CIC condition, namely that the distribution of unobserved types within each group is stable over time, and not additive parallel trends on some arbitrary scale. Athey and Imbens' Kentucky example is the reason this is the right target when scales are not comparable: the additive assumption has no scale-free meaning there.

Research Program

The work has three parts.

The first is the statistic and its inference. For continuous outcomes we derive the asymptotic variance of $\Delta_{s,t}$ from the CIC influence function and validate a site-clustered bootstrap. Ordinal and coarsened outcomes, such as bleeding grades, are the clinically central case. For these, Athey and Imbens show that CIC only bounds the counterfactual distribution. We therefore define lower and upper rank deviations $[\Delta^-_{s,t}, \Delta^+_{s,t}]$ and adopt a conservative rule: equivalence is declared only if the whole interval passes. This treats coarse scales honestly, since a site with a three-level scale contributes wide intervals and little evidence, instead of being silently pooled with fine-grained sites.

The second is staggered adoption. Here we run the test separately for each adoption cohort, using only not-yet-treated units as controls. This avoids the forbidden comparisons that bias two-way fixed effects (Choosing an analytic approach: Key study design considerations in state policy evaluation), and we aggregate across cohorts in the random-effects layer.

The third is a diagnostic decomposition. When the pooled test fails, we report the share of failure due to the common component $\mu_t$, which signals genuine confounding, and the share due to $\tau_s$, which signals site-specific drift. Site-specific drift is often a sign that one site changed its definition partway through the panel. That is exactly the measurement heterogeneity the clinical literature describes, and it should be flagged rather than averaged away.

The assumptions are explicit. Each site's measurement map must be stable over time. Within a site, treated and control units must share the same map. CIC monotonicity must hold within site. The test is not designed to survive a site changing its scale mid-panel; it is designed to detect that event through $\tau_s$.

Baselines

The naive reference point is the conventional event-study screen. It pools all sites with site and time fixed effects, regresses on lead indicators, and declares parallel trends if no lead is significant at 5%. We run it on exactly the same simulated and real panels. It is the field default and serves as the floor. The research file documents the power trap that makes it unsafe (Validating the Parallel Trends Assumption).

The strongest existing method is fect TOST equivalence testing on imputation residuals (A Practical Guide to Counterfactual Estimators for Causal Inference with Time-Series Cross-Sectional Data). We run it with FEct and IFEct counterfactuals and include site-by-time interactions so that it has every fair chance to absorb site differences. Because its bound lives in outcome units, we give it a strong version of $\delta$: the raw-scale bound implied by our percentile bound under the true data-generating process. That information is unavailable in practice, so this setting is generous to the baseline. No reported result found on Lacuna for the size or power of fect TOST under site-specific measurement transformations. The file reports only that IFEct reversed a sign on the UK grants data, with no numbers. We beat it if, at matched power under latent parallel trends, our false-reassurance rate stays within nominal level across transformation families where fect TOST does not.

A third baseline z-scores each site's outcome before running fect TOST. This is the most plausible cheap fix, and it is discussed below. A fourth, oracle baseline maps every site onto the true latent scale and then runs TOST. It is available only in simulation and sets the ceiling on achievable power.

Two ablations isolate our ingredients. The first keeps the rank statistic but pools with a fixed-effect model, which removes $\tau_s$; this shows whether modeling heterogeneity matters for coverage. The second runs CIC without equivalence testing, using a significance test on $\Delta$; this shows what the TOST layer contributes.

Why not just…?

Why not just z-score the outcome within each site and use existing TOST? This is the real threat. If every site's scale is an affine rescaling of a common latent outcome, per-site standardization removes it, and our method should merely tie. The proposal earns its place only if site maps are nonlinear or coarsening. Clinical scales of the kind documented for bleeding grades and implantation-failure definitions are plausibly of this type. To settle it, the simulations sweep the transformation family from affine to strongly convex to ordinal coarsening with site-specific cutpoints. If z-scoring matches our method beyond the affine case, we will report that and treat the rank construction as unnecessary.

Why not just harmonize codes with a toolkit like PyHealth and then test as usual? Code mapping such as AutoMap in PyHealth resolves vocabulary differences, for instance ICD-9 against ICD-10. It does not resolve two valid scales that place cutpoints at different severities, because no deterministic crosswalk exists between them. Our oracle-harmonization baseline shows how much is lost when a crosswalk does not exist. Our method should approach the oracle without needing one.

Why not just log-transform, or run linear DiD within each site and meta-analyze? Athey and Imbens show that additive conclusions flip between levels and logs on the Kentucky workers' compensation panel (Identification and Inference in Nonlinear Difference-In-Differences Models). Within-site linear DiD inherits this: latent parallel trends become non-parallel on any site scale that is nonlinear in the latent outcome. Our experiment on that panel tests whether our verdict is identical under levels and logs, which it must be by construction, while the linear alternatives' verdicts change.

Proposed Experiments

The primary testbed is semi-synthetic. We generate state-level mortality panels with the OPTIC.simRM simulator, built on National Vital Statistics System and Bureau of Labor Statistics data (Moving beyond the classic difference-in-differences model: a simulation study comparing statistical methods for estimating effectiveness of state-level policies). That study reports Type I error as high as 67% for standard models on these panels, which shows how fragile default inference is here. States are grouped into pseudo-sites. Each site receives its own increasing transformation or ordinal coarsening, and a known latent pre-trend violation of controlled size is injected under both instantaneous and staggered adoption.

The primary metric is the false-reassurance rate: how often equivalence is declared when the latent violation exceeds $\delta$. Secondary metrics are power under true latent parallel trends and the bias of the downstream CIC effect estimate conditional on passing the test.

A clinical testbed uses eICU, a multi-hospital ICU database supported in PyHealth. Hospitals act as sites, placebo "policies" are assigned to random hospital-year cohorts, and violations are induced by selecting treated hospitals on pre-period outcome drift. Hospital-specific rescorings of a severity outcome mimic scale heterogeneity. This tests the method under real between-site variation.

The invariance check uses the Kentucky panel, where our verdict must not move between levels and logs. As a sanity check we use California Proposition 99, where Synthetic Difference-in-Differences reports estimates of −27.3 for DID, −19.6 for synthetic control and −15.6 for synthetic DiD, in packs per capita. Our test's pre-period verdict should be consistent with the preference for reweighted comparisons that those estimates suggest.

The decisive result is a curve of false-reassurance rate against transformation nonlinearity. The proposal succeeds if our test holds nominal level across the sweep while fect TOST and z-scored TOST drift upward once site maps become nonlinear or coarsened. It fails, and should be abandoned, if z-scoring tracks it throughout or if the coarsening bounds make it powerless at realistic sample sizes.