Telling Prior Shift from Meaning Shift with Certified Discrimination Bounds

When a word-sense model trained on general text is moved into a technical domain, its accuracy collapses. Supervised systems that reach 85–87% precision in-domain fall to 53–62% when trained on general corpora (SemEval-2010 Task 17). That collapse has two very different causes, and they need different fixes. Sometimes the senses still look the same in embedding space and only their frequencies have changed: in environmental text, the technical sense of expression or conservation is simply more common. Re-estimating the sense priors, which is cheap, repairs this. Other times the contexts in which a sense appears have moved to new directions in embedding space, and no reweighting of the old decision rule can recover them. Repairing that requires new labels or retraining. Machine translation research already separates these two failure types by hand. Measuring Machine Translation Errors in New Domains calls them SCORE and SENSE errors and finds SENSE errors as damaging as unseen words. What is missing is a per-word, annotation-light measurement that says which failure is happening before anyone spends money on adaptation.

We propose to compute that measurement with tools from quantum information that are well understood there and almost unused in NLP. For each sense of a word, we represent its contexts as a density matrix: the average outer product of normalized, low-dimensional contextual embeddings, which is a positive semidefinite matrix with unit trace. Choosing a sense then becomes a state-discrimination problem. A semidefinite program, an optimization over positive semidefinite matrices that has a certified dual, gives the best possible accuracy of any readout in this class under any chosen sense priors. Restricting that program to readouts that are diagonal in the source domain's basis gives the best accuracy achievable by reweighting the old representation alone. The difference between the two optimal values is an operational coherence quantity in the sense of the accessible-coherence literature: the advantage that measurements "coherent" with respect to the source basis have over "incoherent" ones. Our central claim is that this gap, computed per word from a small seed set, predicts how much of the domain loss can be recovered only by retraining. Separately, the change in the restricted optimum under re-estimated priors predicts how much prior correction alone recovers.

Core Claim

Fix a word with senses $s=1,\dots,K$. Let $\phi(x)\in\mathbb{R}^k$ be a context's embedding after projecting it to $k$ dimensions (with $k$ around 32–64, via principal components of that word's pooled contexts) and normalizing it to unit length. The target-domain sense state is $\rho_s = \frac{1}{N_s}\sum_{x\in s}\phi(x)\phi(x)^\top$, estimated from a seed set of $N_s$ labeled target examples. Given priors $p_s$, the certified ceiling is

$$ P^\star(p,\rho)=\max_{M_s\succeq 0,\ \sum_s M_s=I}\ \sum_s p_s \operatorname{Tr}(M_s\rho_s), $$

where the $M_s$ are the measurement operators, one per sense. The dual, which minimizes $\operatorname{Tr}(Y)$ subject to $Y\succeq p_s\rho_s$ for every sense, certifies that no readout in the class exceeds $P^\star$. This class contains every classifier whose score is linear in the second-moment features $\phi\phi^\top$, which includes every quadratic decision rule on the projected embedding. We make no claim of quantum advantage. The object is simply a well-posed, certifiable family of quadratic classifiers.

Let $B$ be the eigenbasis of the pooled source-domain (SemCor) states for the word. Restricting the $M_s$ to be diagonal in $B$ turns the program into a linear program; call its optimum $P_B(p,\rho)$. The basis-mismatch gap is

$$ \Delta = P^\star(\hat p_{\text{tgt}},\rho) - P_B(\hat p_{\text{tgt}},\rho), $$

where $\hat p_{\text{tgt}}$ are target priors re-estimated from unlabeled background text. $\Delta$ is zero exactly when no readout in the class gains anything from directions that cut across the source eigenbasis. This is the case, for instance, when the target states commute with that basis.

The prior-shift share is $P_B(\hat p_{\text{tgt}},\rho)-P_B(p_{\text{src}},\rho)$, the gain available by changing priors while keeping the source geometry. Our hypothesis is that, across words, the prior-shift share tracks the realized gain from prior-correcting a deployed source model, and $\Delta$ tracks the additional gain obtained only by fine-tuning on target data.

Where the Pieces Come From

Prior work plays three roles here: it motivates the gap, supplies the methods, or constrains the design.

Motivation. The motivating evidence comes from the domain-shift papers. SemEval-2010 Task 17 documents the general-to-domain collapse and shows that a seed set of about 100 tagged examples (CFILT) beats the most-frequent-sense baseline. Measuring Machine Translation Errors in New Domains shows that SCORE errors make up 25% of errors in the Subtitles domain while SENSE errors dominate elsewhere. That paper diagnoses with alignment tools and table enhancement after the fact. We aim for an intrinsic per-word quantity computed before adaptation.

Methods. The method comes from two places. The first is the primal-dual formulation of state discrimination summarized in the file. The second is the resource-theory view in which a measure's value is the advantage of resourceful over resourceless operations in a discrimination task. Projective robustness for quantum channels and measurements proves such a ratio for measurements in combined discrimination and exclusion tasks. Set Coherence shows that a set of states' resistance to simultaneous diagonalization sets the advantage in subchannel discrimination. Inequivalent multipartite coherence classes defines accessible coherence as a monotone under restricted incoherent operations. We borrow the operational reading rather than any specific closed-form measure. "Incoherent" becomes "diagonal in the source basis," and the advantage becomes $\Delta$. Set coherence then has a natural reading as the basis-free version of the same question. If the target sense states have zero set coherence, some common basis exists. A learned rotation followed by a diagonal reweighting would then suffice, which is a cheaper fix than full retraining. We compute it as a secondary diagnostic.

Closest prior work and constraints. Three lines of work put language or classification data into density matrices, and they also mark what we avoid.

  • Quantum State Discrimination for Supervised Classification builds per-class average density matrices and discriminates them. It uses the closed-form Pretty Good Measurement rather than the optimal program and evaluates only tabular data. It also shows that the Helstrom bound correlates with accuracy (Pearson ≈ 0.9468). We take that correlation as encouragement that discrimination ceilings track real accuracy. We take its "curtain of dimensionality" from tensor copies as a constraint, so we use no tensor copies and keep $k$ small.
  • A Quantum-Theoretic Approach to Distributional Semantics represents words as density matrices but only compares them by trace overlap.
  • QR-WSD and Q-VWSD use superposition-inspired neural scoring to improve long-tail and visual disambiguation.

None of these separates prior shift from geometric shift, computes a certified ceiling, or addresses technical-domain transfer. We are not proposing a better disambiguator, and the work stands or falls on whether the diagnostic predicts adaptation outcomes.

Research Program

Representations and baseline model. We first build the representations. Contextual embeddings come from a frozen pretrained encoder applied to SemCor (source) and to the English environment-domain text of SemEval-2010 Task 17 (target). For each polysemous target lemma we compute a word-specific projection, source and target sense states, and the source eigenbasis $B$. Target sense states use a seed set in the CFILT regime of roughly 100 tagged examples for the whole domain. We draw the seeds from held-out annotated domain text, disjoint from the test words' evaluation instances. Target priors are re-estimated without labels by expectation-maximization over the 2.7 million words of unannotated background text, using the source-trained classifier's posteriors. The deployed model is a SemCor-trained probe on the same frozen encoder.

Solving the programs. The semidefinite and linear programs have $K$ matrices of size $k\times k$ per word. A standard interior-point solver handles them in well under a second, so the whole vocabulary is cheap and no scalable-solver research is needed. We check strong duality numerically on every word. Because the seed set is small, we bootstrap the seeds to put confidence intervals on $P^\star$, $P_B$, and $\Delta$. We report how many seeds per sense are needed before the interval on $\Delta$ excludes zero.

Ground truth. For each word we measure three realized accuracies on held-out target instances: the source model as deployed, the source model with re-estimated priors, and the source model fine-tuned on the seed set plus a larger annotated target pool where SemEval provides one. The realized prior gain and the realized fine-tuning-only gain are the targets that the two predicted shares must track.

Baselines

Do nothing. The naive reference is the general-domain most-frequent-sense heuristic, which scores 0.505 recall on the English all-words environment task, and the unadapted source probe. The strongest existing domain result in the file is CFILT-2's 0.555 recall on the same task (SemEval-2010 Task 17), obtained with a seed set of about 100 examples. The knowledge-based Kyoto system, which runs Personalized PageRank over WordNet with distributional thesauri, reached 0.52–0.53 in Dutch and Italian. We reproduce an equivalent seed-supervised probe with the same encoder and seed budget so that adaptation numbers are comparable.

Use of the diagnostic. Our system is not meant to beat these numbers by itself; its job is triage. The applied test gives a fixed annotation budget, enough to fine-tune only a fraction of lemmas, and allocates it by ranking lemmas on $\Delta$. All other lemmas receive prior correction only. Every allocation policy gets the same encoder, seed pool, and hyperparameter search. Beating the baselines means two things. First, $\Delta$-ranked allocation must give higher all-words recall than allocation by lemma frequency, by observed source-model error on seeds, or at random. Second, the combined system must reach or exceed CFILT-2's 0.555 recall while annotating fewer lemmas.

Competing shift measures. These are the strongest competitors to the diagnostic itself. The first is a direct probe: train a classifier on the target seeds and use its accuracy, or its gain over the source model, as the predictor. This is the obvious thing a practitioner would do. The second is a distribution distance between source and target embeddings of the word, either a kernel two-sample statistic or a fitted-Gaussian KL divergence. The third is Blacoe-style trace overlap between source and target sense states. All three see exactly the same seeds and unlabeled text as our measure.

Ablations. Two ablations isolate the key ingredients. Replacing the optimal measurement with the Pretty Good Measurement, as in Giuntini et al., tests whether optimality under skewed priors matters. We expect this approximation to hurt most on lemmas with a rare technical sense. Replacing the source eigenbasis with a random orthogonal basis tests whether $\Delta$ is specifically about the source geometry. If that substitution leaves the predictive power unchanged, the coherence reading is empty.

Why not just…?

Why not just use a classical classifier or linear program instead of a semidefinite program? The diagonal-basis linear program is part of the design: $P_B$ is exactly that classical restriction, and the claim lives in the gap between it and the unrestricted program. The semidefinite program here is just a certified maximum over quadratic readouts. We do not claim it disambiguates better than logistic regression, and our end-task numbers will come from ordinary probes. The quantum vocabulary earns its place only by giving a clean way to compare "readouts allowed to rotate" against "readouts tied to the old basis," with a dual certificate on the first.

Why not just train a probe on the target seeds and see how much it helps? This is the most serious threat. A seed-trained probe gives a noisy lower bound on achievable accuracy, and with roughly 100 seeds spread across many lemmas it may be too noisy to rank words. The semidefinite program gives an upper bound on the same data, and the two together bracket what retraining can achieve. The probe also does not by itself separate prior-correctable error from geometric error, whereas our decomposition does by construction. If the direct-probe baseline ranks lemmas for allocation as well as $\Delta$ does, we will report that the coherence quantity is descriptive rather than useful. The allocation experiment is designed to settle this.

Why not just measure distribution shift with a standard divergence? A divergence fires whenever anything changes, including pure prior shift, which is cheap to fix. Our hypothesis is that divergence correlates with total loss but not with the fine-tuning-only gain once prior correction is applied, while $\Delta$ correlates with the latter. A partial-correlation test, controlling for the prior-shift share, decides this.

Why not just use the Pretty Good Measurement, which needs no solver? At this matrix size the solver costs nothing, and the approximation's known weakness is skewed priors, which is precisely the domain-shift regime. The ablation measures how much the ranking degrades, and if it does not, we will recommend the closed form.

Decisive Checks

The primary test is per-lemma rank correlation on the SemEval-2010 English environment task. We correlate the predicted prior-shift share with realized prior-correction gain, and $\Delta$ with realized fine-tuning-only gain, with bootstrap intervals over seeds. The hypothesis fails if $\Delta$'s partial correlation is no better than the direct-probe or divergence baselines, or if the random-basis ablation matches it.

The secondary test is the budgeted allocation experiment against the 0.555 CFILT-2 reference. Here $\Delta$-guided allocation must match that recall while annotating fewer lemmas.

A third, cross-task check maps the decomposition onto the translation taxonomy. Using the Hansard-trained French–English setting of Measuring Machine Translation Errors in New Domains, we build states over a source word's translation choices. We then check whether the measure, aggregated over words, calls Subtitles prior-dominated, consistent with its reported 25% SCORE errors, and calls Science and EMEA geometry-dominated, consistent with their large SENSE effects. If the ordering of domains comes out wrong, the decomposition is not capturing what translation researchers mean by these error types, and we would say so.