Observational studies in medicine increasingly try to reduce residual confounding by adjusting for what is written in clinical notes, not only for what is recorded in structured tables. Residual confounding is the bias that remains after standard adjustment, and it arises when something that drives both treatment and outcome is missing or only coarsely recorded. The usual recipe is to pass each note through a pretrained language model, take the resulting vector (the "representation"), and use it as a covariate in a flexible causal estimator. The instinct is then to reach for the best domain-specific model, meaning a model pretrained on biomedical or clinical text rather than on web text. This proposal argues that this instinct is only half right. A domain model captures more of the hidden clinical state that confounds the comparison. It also captures more of the incidental signals that predict who gets treated without affecting the outcome, such as care unit, note author, or documentation style. Adjusting for that second kind of signal shrinks the usable variation in treatment and can amplify whatever confounding remains.
We propose to measure these two quantities directly for general and domain-pretrained encoders on semi-synthetic MIMIC notes with known ground truth. We will test whether the two quantities, rather than downstream predictive accuracy, explain residual bias. We then test a simple fix: a low-dimensional projection of the domain representation, trained to predict the outcome, that keeps the confounder information and discards the treatment-only information. If the story holds, practitioners should choose and shape a clinical encoder for text-based adjustment using a confounding-specific diagnostic, not a leaderboard.
Core Claim
Suppose the outcome depends on an unmeasured condition $U$ that is mentioned in the note, and we adjust using the note's representation $R$ in place of $U$. In the partially linear model, the large-sample bias of a double machine learning estimate is
$$ \text{bias} \approx \frac{\mathbb{E}\big[\mathrm{Cov}(A,\, g(U) \mid R, X)\big]}{\mathbb{E}\big[\mathrm{Var}(A \mid R, X)\big]}, $$
Here $A$ is the binary treatment, $X$ the structured covariates, and $g(U)$ the part of the outcome driven by the hidden condition. Double machine learning is an estimator that fits separate flexible models for treatment and outcome and combines their errors so that small fitting mistakes do not bias the effect estimate.
The numerator is the confounding that $R$ fails to absorb, and it shrinks as the representation encodes more of $U$. The denominator is the treatment variation left after adjustment. It shrinks whenever $R$ encodes treatment-predictive signals unrelated to the outcome, and these behave like instruments, variables that shift treatment without directly affecting the outcome. Domain pretraining plausibly pushes both terms down. The numerator shrinks because the encoder separates "afib" from colloquial noise. The denominator shrinks because it also sharpens stylistic and institutional cues that forecast treatment. The claim is that the net effect on residual bias is not monotone in how "good" the domain model is. We further claim that an outcome-anchored projection can recover the numerator gain without paying the denominator cost.
Where This Sits
Three recent papers connect text, pretrained models and hidden confounding, and all three leave the representation's content unexamined. Adjustment for Confounding using Pre-Trained Representations supplies our estimator. It shows that sparsity-based learners such as the Lasso break on pretrained features because those features have no preferred axes. It also proves that neural-network nuisance models in double machine learning adapt to the low intrinsic dimension of the features. In their chest X-ray example that dimension is about 12, against an ambient dimension of 1024. Their work uses general BERT and DenseNet as fixed extractors with synthetic treatment and outcome, and it asks whether the estimator converges. We ask what the extractor should encode.
Proximal Causal Inference With Text Data motivates the problem and supplies our strongest competitor and our semi-synthetic design. It builds MIMIC-III benchmarks in which a real condition (atrial fibrillation, hypertension) plays the hidden confounder. It shows that plugging zero-shot LLM labels into ordinary adjustment stays biased, and it fixes this with a proximal estimator that requires two separate notes per patient. It used only general LLMs and a linear estimator. Mitigating Hidden Confounding by Progressive Confounder Imputation via Large Language Models has general LLMs hypothesize and impute missing confounders for tabular benchmarks. Its authors flag niche clinical text as open. We do not generate confounders; we audit what an encoder already carries.
The older review Text and Causal Inference constrains the design. It warns that text representations frequently break overlap, meaning the requirement that similar patients have a real chance of either treatment. It also warns that arbitrary preprocessing choices swing estimates. Our denominator term is a precise version of the overlap warning, and our fixed protocol across encoders is a response to the preprocessing warning.
On the pretraining side, Revisiting Clinical Outcome Prediction for MIMIC-IV finds that PubMedBERT, trained on literature with its own vocabulary, beats models pretrained on MIMIC-III notes for clinical prediction. BioBART finds that literature-only pretraining falls short on informal clinical text. This contradiction matters here. A notes-pretrained encoder may be worse at prediction yet more attuned to the documentation habits that forecast treatment, and our decomposition can tell which effect dominates. BERTHop similarly reports that in-domain pretraining improves predictive accuracy. All of this evidence concerns prediction, which is exactly the criterion we argue is wrong for adjustment.
Research Program
The first component is a measurement protocol for any frozen encoder. It has two parts. Confounder capture is the fraction of variance in $U$ that a held-out neural probe recovers from $R$ given $X$. Treatment-only capture is the fraction of variance in a known outcome-irrelevant but treatment-relevant note attribute $V$ that the same kind of probe recovers. Candidate attributes are care unit, note category, and caregiver identifier, all available as MIMIC metadata. We also estimate each encoder's intrinsic dimension, following Adjustment for Confounding using Pre-Trained Representations, since low dimension governs finite-sample behavior. The encoders compared are general BERT, BioBERT, ClinicalBERT, PubMedBERT, and a mean-pooled OLMo-7B embedding as the general-LLM reference. All are run with identical pooling, truncation and note selection.
The second component is the semi-synthetic benchmark. We follow the Chen et al. construction on MIMIC-III and extend it. The hidden $U$ is a real documented condition. Treatment is simulated from $U$, $X$ and a tunable dose of $V$. The outcome is simulated from $U$, $X$ and $A$ with a known effect. Sweeping the strength of $V$ in treatment assignment lets us test the denominator prediction directly: the gap between general and domain encoders should shrink or reverse as treatment-only signal strengthens.
The third component is the fix. We train a $k$-dimensional projection of the domain representation, with $k$ near the measured intrinsic dimension, to predict the outcome within the control arm only. The treatment model then sees only this projection plus $X$. The motivation is the classical rule of adjusting for outcome predictors rather than treatment predictors. What is new is applying it to a rotation-invariant neural representation, where variable selection is impossible, as Schulte et al. show. To keep the outcome model honest we use cross-fitting, so the projection is learned on folds disjoint from those used for estimation.
The fourth component is a real-data negative control. We use albumin plus crystalloids versus crystalloids in sepsis on MIMIC-IV, where Step-by-step causal analysis of EHRs to ground decision-making recovered the randomized-trial null with careful target-trial design. Adding note representations should keep the estimate near null and tighten it, not push it away.
Baselines
The naive reference is double machine learning with neural nuisance models on structured covariates $X$ alone, ignoring notes. It sets how much residual bias the text must remove. An oracle that adjusts for the true $U$ sets the floor. The simplest text baseline is ordinary adjustment on zero-shot LLM labels of $U$, as run in Proximal Causal Inference With Text Data. That paper reports it "remained significantly biased," but the file gives no numeric bias, so no reported result was found on Lacuna for its exact value, and we will rerun it.
The strongest representation baseline is the double machine learning estimator with neural nuisance models of Adjustment for Confounding using Pre-Trained Representations, applied to each encoder. Their paper reports near-unbiasedness and valid normal approximations only graphically, so no reported numeric result was found on Lacuna for its bias or coverage. The strongest method using a different identification strategy is the two-note proximal g-formula of Chen et al., run with both a general LLM and a domain encoder, on the subset of patients with two eligible notes.
For fairness, every method uses the same patients, the same simulated draws (500 replicates per configuration), the same five-fold cross-fitting, and the same hyperparameter search budget per nuisance model, tuned on predictive loss only. The one predictive number the file offers for domain encoders on MIMIC-IV is PubMedBERT's 86.86% AUROC and 17.24% PR-AUC on emergency department diagnosis prediction (Revisiting Clinical Outcome Prediction for MIMIC-IV). We use it only to confirm our encoders are loaded and pooled sensibly.
Two ablations isolate the key ingredient. One is an unsupervised projection (principal components to the same $k$). The other is a projection trained to predict treatment rather than outcome, which should do worst if the denominator mechanism is real. Beating the baselines means lower absolute bias and root-mean-squared error than the best raw-encoder estimator, with 95% interval coverage within two points of nominal, and with the advantage growing as treatment-only signal strengthens.
Why not just…?
Why not just use the best clinical encoder and stop? This is the central comparison, not a straw man. If PubMedBERT or ClinicalBERT raw representations already give the lowest bias across the whole treatment-only sweep, then the denominator effect is negligible in practice. In that case we would report the measurement protocol as a selection tool and drop the projection. The prediction that distinguishes our account is a crossover. At weak treatment-only signal, domain encoders beat general ones. At strong signal, the ranking narrows or flips, and the flip is tracked by treatment-only capture rather than predictive AUROC.
Why not just use proximal inference with two notes? Chen et al. show it removes the bias that naive label adjustment leaves. However, it needs two independent pre-treatment notes, a binary confounder and a linear second stage, and it can halt when its odds-ratio check fails. We run it head to head on the two-note subset. If it matches our estimator there, its advantage is identification without assuming the note fully captures $U$, while ours applies to single-note patients and continuous states. The two estimators may be complementary rather than competing, and we will report where each fails.
Why not just trim extreme propensity scores or use overlap weights? This is a real threat, because trimming directly attacks the small-denominator problem. However, it changes the target population, and it discards patients rather than discarding the information that made them look extreme. We include trimmed and overlap-weighted versions of the raw-encoder estimator. If they achieve equal bias and coverage for the full-population effect, which trimming generally cannot target, the projection adds nothing and we will say so.
Why not just fine-tune the encoder end to end on the outcome? Full fine-tuning is the heavy version of our projection, and it risks overfitting nuisance models in a way that breaks the double machine learning guarantees. This is a gap Adjustment for Confounding using Pre-Trained Representations leaves open. We run a cross-fitted fine-tuned variant at matched compute. If it beats the projection without losing coverage, it becomes the recommended recipe, and the decomposition still explains why.
Decisive Checks
The account succeeds if three things hold across the MIMIC-III sweep. First, confounder capture and treatment-only capture jointly predict each encoder's residual bias much better than its predictive AUROC does. Second, the domain-versus-general ranking changes as treatment-only strength increases. Third, the outcome-anchored projection lowers bias relative to the best raw domain encoder, while the treatment-anchored projection raises it.
The account also has to survive the real-data control. On albumin in sepsis, text-augmented estimates must stay consistent with the randomized null reported in Step-by-step causal analysis of EHRs to ground decision-making. A raw notes-pretrained encoder that drifts away from null while the projection does not would be the clearest real-world sign of the mechanism. If instead capture of treatment-only signal barely moves across encoders, the proposal's core concern is empirically small, and the contribution reduces to a validated protocol for choosing encoders.
References
- Adjustment for Confounding using Pre-Trained Representations
- Proximal Causal Inference With Text Data
- Mitigating Hidden Confounding by Progressive Confounder Imputation via Large Language Models
- Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates
- Revisiting Clinical Outcome Prediction for MIMIC-IV
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
- Step-by-step causal analysis of EHRs to ground decision-making