Language models pre-trained on biomedical and clinical text, such as BioBERT, learn the specialized meaning of words like "culture" or "arrest." They still tend to judge two sentences as similar when the sentences share many words. In a clinical note, many words are always shared: "patient," "mg," "history," and the same templated phrasing. So "no evidence of pneumonia on left" and "evidence of pneumonia on left" look nearly identical to the model. A cheap, well-known fix for this kind of surface similarity in general-domain embeddings is common component removal. It subtracts the average vector and projects away the few directions that carry most of the variance. In general text these directions mostly encode word frequency and boilerplate. We propose to test whether this fix carries over to clinical encoders. We also propose a refinement for the case where it does not. The refinement uses automatically generated, meaning-changing edits (inserted negations) to mark which dominant directions carry clinical meaning, and keeps those directions instead of deleting them.
The core claim fits in two sentences. In domain-pretrained encoders, the dominant directions of the embedding space mix template and frequency signal with real clinical meaning such as negation and polarity, so blindly removing the top components should help less, or even hurt, compared with general text. Removing only the high-variance directions that negation-style edits do not move should lower the model's reliance on word overlap on clinical similarity and inference benchmarks. It should do so without labels, external entity taggers, or retrieval indexes. The negation edits are used only as probes and are never used for training.
What is known, and what is missing
Three general-domain results supply the method and the motivation. All-but-the-Top centers static word vectors and removes roughly $d/100$ top principal components, reporting about 4% average gains on STS 2012–2015. STS here means semantic textual similarity: sentence pairs scored 0–5 by humans. SIF removes the first principal component of frequency-weighted sentence averages. Follow-up critiques (Evaluation of Unsupervised Compositional Representations, A Critique of SIF) find that the removal step does much of the work. BERT-flow shows that BERT sentence embeddings sit in a narrow cone, a property called anisotropy. In that geometry, cosine similarity tracks string edit distance. A learned invertible map to a Gaussian reduces that correlation from −50.49 to −28.01 and lifts STS by 8.16 Spearman points on average. All three studies use general-domain encoders and general-domain benchmarks.
Two newer methods connect component removal to surface-token shortcuts, again only on general benchmarks. 3R subtracts an LLM-generated "high-frequency noise" embedding along low-variance dimensions during SimCSE training, raising SimCSE-BERT-base average STS from 75.21 to 77.52. SimCSE is contrastive training that treats two dropout views of one sentence as a positive pair. Its noise pool is built from generic high-frequency words, not from clinical templates. PURE removes the top component of each input's token embeddings during fine-tuning, cutting the adversarial performance drop rate from 81.7 to 58.4. Both methods remove the top direction without checking what it encodes.
Two results constrain the design. LIR removes the top language-identity component from multilingual encoders. It raises XQuAD-R retrieval MAP from 23.3 to 45.2. However, it degrades English and strongly aligned models, where the top component already carries meaning. This is the failure mode we expect in clinical encoders, where negation and normal-versus-abnormal findings are frequent enough to dominate variance. LIR also reports that components estimated on general text transfer to product reviews. That finding is in tension with the "drowning effect" described for domain models (Durrani et al., MonoCoder), and we test it directly.
On the domain side, two papers motivate the work. MedNLI Is Not Immune finds that physician-written clinical inference data contains lexical artifacts. A hypothesis-only classifier reaches 62.6% against a 33.3% majority baseline. Full models fall from about 51% to 40% on a harder split filtered with AFLite, an adversarial filtering procedure that removes easily predicted examples. The authors offer no fix at the representation level. EARA documents an "entity penalty": BioBERT and BlueBERT error rises on clinical pairs with more than five entities. EARA fixes this with cTAKES entity alignment, an attention loss, and MIMIC-III retrieval, reaching 0.887 Pearson on N2C2STS. Neither paper asks whether the shortcut lives in a few dominant directions of the domain embedding space.
Core mechanism
Let $C$ be the covariance of centered sentence embeddings computed on unlabeled in-domain text (MIMIC-III notes or PubMed sentences). We generate meaning-altering minimal edits of the same sentences automatically, using only negation insertion and removal with simple cue rules. Let $E$ be the covariance of the difference vectors between each sentence and its edited version, rescaled to match the trace of $C$. We remove the top $k$ solutions of the contrastive principal component problem from Contrastive PCA:
$$v^\star = \arg\max_{\|v\|=1} \; v^\top C v - \alpha\, v^\top E v,$$
Here $\alpha \ge 0$ trades off how much variance a direction holds against how much the edits move along it. When $\alpha = 0$, this reduces to All-but-the-Top on domain text. The idea is that directions with high corpus variance but low edit sensitivity are template and frequency scaffolding, and removing them should not cost meaning. The method needs one eigendecomposition and no trained parameters. It applies after the encoder for unsupervised similarity, or inside the encoder at the pooling step, PURE-style, for fine-tuned inference.
Negation is the only edit family used to build $E$. The evaluation deliberately includes edit families the method never sees: dosage and number swaps, laterality swaps (left/right), and subject–object or drug–disease argument swaps. This separation is what lets us distinguish "learned a general meaning-preserving geometry" from "memorized negation."
Research Program
The first step is diagnostic. For BERT-base, BioBERT, BlueBERT and PubMedBERT, we measure three things on in-domain text. The first is anisotropy: the average cosine between random sentences and the share of variance in the top components. The second is an overlap-conflation index. Following BERT-flow, it is the partial correlation between pair cosine and token Jaccard overlap on N2C2STS and EBMSASS, controlling for gold similarity. The third is the edit energy of each top component, meaning how much of $E$ falls on it. The prediction is that domain encoders show anisotropy comparable to or stronger than general BERT, and that at least one of their top few components carries substantial negation energy, unlike general BERT on general text. If no top component carries edit energy, plain removal should suffice, and the paper becomes a clean negative result about the refinement.
The second step answers the open question from LIR about where the components should be estimated. We compare components estimated on Wikipedia, on clinical notes, and contrastively against Wikipedia, with general text as the background.
The third step is the main comparison on similarity, inference, and contrast sets, described next.
Baselines
The naive reference point is the raw encoder: mean-pooled BioBERT or BlueBERT embeddings scored by cosine, with no post-processing. Every method is applied to the same frozen checkpoints and the same pooled vectors.
The direct ablation of our key ingredient is All-but-the-Top on domain text, which is our objective with $\alpha = 0$. We also run SIF-style first-component removal and LIR-style removal with components estimated on general text. Ours and the ablation receive identical tuning budgets: $k \in \{1,\dots,10\}$, plus $\alpha$ for ours on a fixed grid. Both are tuned only on the N2C2STS training split. Beating the ablation means higher test Pearson on N2C2STS and a lower overlap-conflation index on the held-out contrast families, with the improvement exceeding variance across seeds and bootstrap resamples.
The strongest unsupervised competitor is BERT-flow trained on the same unlabeled clinical text as our covariance estimate. Its reported general-domain result, an edit-distance correlation reduced from −50.49 to −28.01, sets the size of decoupling we should match. We also include ZCA whitening as an equal-cost linear control; we found no reported result on Lacuna for whitening on clinical encoders. SimCSE trained on the same clinical text, with and without 3R, tests whether contrastive training already removes the problem. We then apply our removal on top of SimCSE to test whether the two effects add.
The strongest existing method is EARA, reported at 0.887 Pearson on N2C2STS with roughly 20% lower error than BlueBERT on pairs with more than five entities. EARA is supervised and uses cTAKES and MIMIC-III retrieval, so we do not expect a post-hoc unsupervised method to match its absolute score. The fair comparison is in the fine-tuned setting. We insert our removal at the pooling layer of a BlueBERT fine-tuned on N2C2STS under EARA's training budget and compare against plain fine-tuned BlueBERT, PURE, and EARA's reported figures. We count it a win if our method recovers a large share of EARA's high-entity error reduction relative to BlueBERT without any external tools. For BIOSSES and EBMSASS we found no reported baseline numbers on Lacuna and will run all baselines ourselves.
For MedNLI, the reference is the fine-tuned BlueBERT premise+hypothesis model and its reported collapse from about 51% to 40% on the AFLite hard split. We compare it against PURE and against our in-model removal. Success means narrowing that gap, together with lower accuracy for a hypothesis-only probe trained on the processed representations.
Why not just…?
Why not just remove the top principal components, as All-but-the-Top does? This is the most serious alternative. If the top domain components carry no clinical meaning, it will work as well as our method at lower complexity. LIR's damage to English and to aligned models is the evidence that top components can carry meaning. The diagnostic step measures edit energy per component directly. The $\alpha = 0$ ablation, run under the same tuning budget, decides the question. If the ablation matches ours on the held-out edit families, we will report that plain removal is sufficient for clinical encoders and drop the refinement.
Why not just fine-tune on adversarial or scrambled pairs? PAWS showed that 12,000 such pairs lift BERT from 33% to 85% on its scrambled pairs. But that recovery was measured on the same kind of edits used in training. It also needs fluent, validated pairs, which in medicine means expert review. PURE reports that adversarial augmentation costs 3–4 times the training compute. We run this alternative as a baseline: fine-tune on our synthetic negation edits, then test on the held-out dosage, laterality, and argument-swap families. If augmentation generalizes to unseen edit families better than geometric removal does, that is a real result and we will report it.
Why not just train contrastively on clinical text? 3R reports that SimCSE still assigns 0.86 cosine to distinct sentences that share high-frequency words, and that targeted removal adds 2.31 STS points on top of SimCSE. Clinical templates should make this residual larger. The SimCSE plus removal comparison measures whether this holds in the clinical domain.
Why not just use a normalizing flow? A flow may decouple overlap more completely, because it is nonlinear. It needs training and gives no interpretable account of which directions carried the shortcut. If BERT-flow beats us on the overlap-conflation index at equal data, we will conclude that the shortcut is not confined to a low-dimensional linear subspace. That would be a substantive finding in itself.
Proposed Experiments
The evaluation set combines N2C2STS (1,642 train and 412 test pairs), BIOSSES, and EBMSASS, scored with Pearson correlation, and MedNLI full and AFLite-hard splits, scored with accuracy. We add a new clinical contrast set built from N2C2STS and MedNLI sentences. It has four edit families: negation, which is also the probe family; dosage and number; laterality; and argument swap. A clinician validates a few hundred items per family, for evaluation only. As general-domain controls, we run STS-B and PAWS to check that the method does not damage general behavior. We report each metric together with the overlap-conflation index, stratified by entity count as in EARA.
Three checks decide the proposal. The first is whether domain encoders' top components carry measurable negation energy. If they do not, the refinement has no reason to exist. The second is whether our method beats domain All-but-the-Top on the held-out edit families, not only on negation. The third is whether, inside fine-tuned BlueBERT, it reduces the more-than-five-entity error and narrows the MedNLI hard-split gap more than PURE does. Passing all three would show that a clinical encoder's reliance on word overlap lives largely in a few identifiable directions, and that those directions can be removed without throwing away clinical meaning.
References
- All-but-the-Top: Simple and Effective Postprocessing for Word Representations
- A Simple but Tough-to-Beat Baseline for Sentence Embeddings
- Evaluation of Unsupervised Compositional Representations
- A Critique of the Smooth Inverse Frequency Sentence Embeddings (Student Abstract)
- On the Sentence Embeddings from Pre-trained Language Models
- 3R: Enhancing Sentence Representation Learning via Redundant Representation Reduction
- Adversarial Defense without Adversarial Defense : Enhancing Language Model Robustness via Instance-level Principal Component Removal
- A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations
- A Deep Fusion Model for Domain Adaptation in Phrase-based MT
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domain
- EARA: Improving Biomedical Semantic Textual Similarity with Entity-Aligned Attention and Retrieval Augmentation
- Contrastive Principal Component Analysis
- PAWS: Paraphrase Adversaries from Word Scrambling