Researchers increasingly estimate treatment effects from clinical notes instead of only from structured records. Doctors give treatments to the patients who need them most, and that "indication" (the frailty, the pain, the bedside sense that the patient is crashing) is often written only in free text. If the text is ignored, the treatment looks harmful merely because sicker patients received it. This is confounding by indication. Recent pipelines therefore ask a language model to read the notes and recover the hidden severity signal. These include proxy extraction with zero-shot LLMs, regression on doubly robust targets from synthetic narratives, and balancing in a low-dimensional subspace of LLM embeddings. Language models, however, are notoriously sensitive to small, meaning-preserving edits to their input. No one has asked what that sensitivity does to the causal estimate rather than to classification accuracy.
The answer turns on a simple point that the robustness literature does not cover. A perturbation harms an effect estimate only when the extraction errors it causes differ between treated and untreated patients. That is exactly how real clinical documentation varies: the deteriorating patient who gets the emergency drug also gets the hurried, abbreviated, typo-ridden note. We propose to measure the worst-case shift in an estimated treatment effect over a budget of meaning-preserving edits to the notes. We expect that a perturbation pattern correlated with treatment arm can reopen the confounding path that the pipeline was built to close, even when average extraction accuracy barely moves. The resulting "perturbation effect interval" is a reportable sensitivity quantity, in the same spirit as classical sensitivity analysis for unmeasured confounding.
Core Claim
A text-based causal pipeline is only as robust as its extraction errors are balanced across treatment arms. Write $T$ for the corpus of pre-treatment notes, $\hat\tau(T)$ for the pipeline's average treatment effect estimate, and $\mathcal{P}_\epsilon(T)$ for the set of corpora reachable by meaning-preserving edits that touch at most a fraction $\epsilon$ of tokens per note. We define
$$\Delta(\epsilon) = \max_{\tilde T \in \mathcal{P}_\epsilon(T)} \left| \hat\tau(\tilde T) - \hat\tau(T) \right|.$$
For a binary severity confounder $U$ extracted as $\hat U$, the standard misclassification argument says that residual bias after adjusting for $\hat U$ is driven by the gap in error rates between arms. Let $s_a$ be the extractor's sensitivity for detecting $U$ in arm $a$, let $\pi_a$ be the prevalence of $U$ in that arm, and let $\beta_U$ be the confounder's effect on the outcome. Then the residual bias scales roughly as
$$\delta = \left( \pi_1 (1 - s_1) - \pi_0 (1 - s_0) \right) \beta_U.$$
This expression comes from treating missed positives in each arm as unadjusted confounder mass. Uniform noise lowers both $s_0$ and $s_1$ together. Under strong channeling, where $\pi_1 \gg \pi_0$, even arm-symmetric degradation biases the estimate, and arm-differential degradation is worse.
Our first claim is that $\Delta(\epsilon)$ at realistic budgets is large enough to change qualitative conclusions, such as the sign of the effect or whether it is significant. Our second claim is that the arm-differential error statistic predicts $\Delta(\epsilon)$ far better than the average accuracy drop that NLP robustness work reports.
Relation to Prior Work
Four papers supply the pipelines we audit, and all of them assume the text is fixed and benign. Proximal Causal Inference With Text Data extracts a treatment proxy and an outcome proxy from two separate pre-treatment notes. It does this with zero-shot FLAN-T5 or OLMo prompts and guards the analysis with an odds-ratio falsification heuristic. LLM-Driven Treatment Effect Estimation Under Inference Time Text Confounding trains a BERT regressor that maps GPT-4o-mini narratives to doubly robust pseudo-outcomes. It tests only formal-versus-colloquial phrasing. Spiked-CFR balances treated and control LLM embeddings in a low-dimensional worst-case subspace found by a continuous minimax game. Two other papers use "adversarial" to mean something different from ours. On the Estimation of Treatment Effect with Text Covariates uses an adversarial discriminator to scrub treatment-only text features. Adversarial De-confounding in Individualised Treatment Effects Estimation uses gradient reversal to the same end on tabular data. In all of this work, adversaries live in representation space as training regularizers. None of it perturbs the documents, and none of it asks whether the estimate survives such perturbations. That is the gap we fill.
The robustness literature motivates the threat. BERT-ATTACK drops fine-tuned BERT on IMDB from 90.9% to 11.4% accuracy with few word changes. Benchmarking Robustness of Machine Reading Comprehension Models reports a 44.5% average drop under distractor extraction, while human accuracy stays near 90%. Diagnosing our datasets shows that clinical jargon is rare in pretraining corpora, which suggests that abbreviation variants are a realistic attack surface. These works measure prediction flips, though, and we argue that flips are the wrong unit for causal work.
Defense work constrains our design. TaiChi and Manifold Purification document that adversarial augmentation degrades clean representations. This is a risk specific to causal pipelines, where faint prognostic signals matter. Text and Causal Inference warns that arbitrary pipeline choices fork causal conclusions. Our interval makes one family of those forks measurable. Finally, the falsification heuristic above and the nuisance-error diagnostics of Uncovering Bias Mechanisms in Observational Studies might misfire when error comes from text sensitivity rather than structural bias. We test that directly.
Research Program
We will build three components.
The first is a clinical perturbation set $\mathcal{P}_\epsilon$ restricted to edits that preserve clinical meaning. It covers keyboard typos, expansion or contraction of standard abbreviations (for example "afib" versus "atrial fibrillation"), reordering of independent sentences, and insertion of non-pertinent boilerplate sentences in the style of AddSent. We exclude open-ended synonym substitution. Masked-language-model substitutions in clinical text can silently change meaning, for example by flipping negation or laterality. A clinician panel will validate a random sample of edits, and edits judged meaning-altering will be removed from the budget.
The second is a causal-target attack. Instead of maximizing label flips, it greedily selects edits that maximize $|\hat\tau(\tilde T) - \hat\tau(T)|$. Candidate edits are scored by their first-order effect on the estimator through the extractor's output. We run it in two regimes. The arm-symmetric regime applies the same budget to every note. The arm-differential regime lets the edit budget depend on treatment. This regime is calibrated, not adversarially chosen: we estimate how typo and abbreviation rates actually differ between treated and untreated patients' notes in MIMIC-III, and use those rates. It answers the realistic question of what happens when documentation quality tracks acuity.
The third is the reporting quantity $\Delta(\epsilon)$ together with the arm-differential error statistic. Both are computed for each pipeline, along with a test of whether the proximal falsification heuristic trips under perturbation.
Baselines
The naive reference point is the clean-text estimate with no perturbation. Next to it we report its error against the known semi-synthetic truth, so that $\Delta(\epsilon)$ can be read relative to the pipeline's ordinary error. A second reference point is random perturbation drawn uniformly from $\mathcal{P}_\epsilon$ at the same budget. This tells us whether a search is needed at all or whether ordinary noise already moves the estimate.
The strongest existing attack is BERT-ATTACK adapted to our meaning-preserving edit set (BERT-ATTACK). It targets the extractor's predictions, and it is the obvious tool a reviewer would reach for. We give it the same edit budget, the same candidate edits, and the same number of model queries as our causal-target attack. Our attack beats it if, at matched budget, it produces a larger shift in the effect estimate. Our predictive claim beats the prediction-level view if the arm-differential error statistic ranks perturbation conditions by $\Delta(\epsilon)$ with clearly higher correlation than the extractor's accuracy drop does. We will preregister these comparisons rather than fix numeric thresholds now.
The strongest existing estimators are the audit targets themselves, and the reported numbers set the scale of what is at stake. TCA reports roughly a 35% PEHE reduction over naive text baselines on MIMIC-III and roughly 40% on the International Stroke Trial (TCA). PEHE is the error in estimating per-patient effects. A key outcome is whether perturbation at small budgets erases that margin. For the proximal pipeline, which reports low bias relative to backdoor adjustment, the file gives no exact figure ("no reported result found on Lacuna for proximal text pipeline bias on MIMIC-III"). The same holds for Spiked-CFR's RMSE-ATE on the text-adapted IHDP and Reddit benchmarks. We rerun all three under their published configurations.
The ablation that isolates our key ingredient compares arm-symmetric and arm-differential budgets at equal total edit counts. If the differential condition does not produce a markedly larger effect shift, our central mechanism is wrong.
Why not just…?
Why not just spell-correct and expand abbreviations before extraction? This is the most serious threat. If canonicalization collapses most of $\mathcal{P}_\epsilon$, the problem is solved cheaply and the paper should say so. We include a medical-dictionary normalizer as a defense arm and report $\Delta(\epsilon)$ after normalization. Normalization cannot undo inserted distractor sentences or reordering, and it can introduce its own errors on ambiguous abbreviations. Whether those residual effects matter for the estimate is an empirical question we settle. It would not be resolved by argument.
Why not just use classical unmeasured-confounding sensitivity analysis? Classical analysis asks how strong an unseen confounder would have to be to explain away the effect. It does not tell an analyst that the confounder they did measure, through text, is measured differently in each arm because of documentation style. Our interval is complementary. We report both, and we test whether $\Delta(\epsilon)$ adds information beyond the classical bound.
Why not just apply adversarial data augmentation to the extractor? TaiChi and Manifold Purification show that augmentation trades away clean accuracy. In a causal pipeline that loss can erase the faint prognostic distinctions the estimator needs. We run augmentation and a TaiChi-style two-branch defense under the same budget, and we measure clean error and $\Delta(\epsilon)$ together. A defense that lowers average sensitivity but leaves the arm-differential statistic unchanged would support our thesis.
Why not just report the extractor's accuracy under noise? This is precisely the claim we test. Our analysis predicts that equal accuracy drops can give very different effect shifts depending on how the errors split across arms. The rank-correlation comparison in the baselines decides it.
Proposed Experiments
We use three benchmarks with known ground truth. The first is the MIMIC-III semi-synthetic proximal setup over four latent conditions such as atrial fibrillation and hypertension, following Proximal Causal Inference With Text Data. The second is the MIMIC-III and International Stroke Trial semi-synthetic setups from TCA. The third is the text-adapted IHDP benchmark from Spiked-CFR. Metrics are the absolute shift in the average treatment effect, PEHE, sign-flip and significance-flip rates, the arm-differential error statistic, and the falsification-heuristic trip rate. We report all of these across budgets $\epsilon$ from 1% to 10% of tokens.
Three checks are decisive. First, under calibrated arm-differential perturbation at small budgets, at least one pipeline's estimate should move by more than its own clean error. Second, the arm-differential statistic should predict effect shifts better than accuracy drop. Third, perturbation alone, with no change to the confounding structure, should cause the odds-ratio falsification heuristic to abort analyses that are in fact identifiable. Failure of the first check, robustly across pipelines and after normalization, would itself be a useful negative result: it would mean current text-based causal pipelines are safer than the robustness literature implies.
References
- Proximal Causal Inference With Text Data
- LLM-Driven Treatment Effect Estimation Under Inference Time Text Confounding
- Spiked-CFR: Causal Representation Learning from LLMs via Wasserstein Projection Pursuit
- On the Estimation of Treatment Effect with Text Covariates
- Adversarial De-confounding in Individualised Treatment Effects Estimation
- BERT-ATTACK: Adversarial Attack Against BERT Using BERT
- Benchmarking Robustness of Machine Reading Comprehension Models
- Diagnosing our datasets: How does my language model learn clinical information?
- TaiChi: Improving the Robustness of NLP Models by Seeking Common Ground While Reserving Differences
- Breaking the Adversarial Robustness-Performance Trade-off in Text Classification via Manifold Purification
- Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates
- Uncovering Bias Mechanisms in Observational Studies