Word sense induction is the task of discovering, from raw text alone, how many distinct meanings a word has ("bank" as riverside versus lender) and which usage carries which meaning. After two decades of methods, a sobering result stands. On text with a natural word distribution, the trivial rule "every word has exactly one sense" beats every fully unsupervised method, and frontier language models do worse still (In the LLM era, Word Sense Induction remains unsolved). The failure mode is over-splitting. Clustering methods carve contextual embeddings into groups whenever the vectors have any lumpy structure, and most of that structure reflects topic, genre and syntax rather than meaning.
We propose to stop treating sense discovery as clustering and to treat it as a question from latent-variable identifiability. That literature asks when a hidden variable is recoverable from the observed variables it causes, often called its "children" or "measurements". Our test is this: a word has $K$ senses only if a $K$-state hidden variable is identifiable from several separate views of its contexts, after accounting for a competing hidden variable (document topic) that also shapes those views. Concretely, the number of senses becomes the rank of a joint probability table over discretized context views, chosen by a statistical test whose default answer is "one". Splitting is something the data must earn. When the structural assumptions behind the test fail, the method abstains and reports one sense rather than inventing clusters.
Core Claim
The claim is that sense counting can be done by tensor-rank model selection over conditionally independent context views, with topic conditioning and an abstention rule. Under natural-distribution evaluation, this should recover the strength of the one-cluster baseline on monosemous words while still splitting genuinely polysemous ones. Existing non-parametric methods cannot get both, because they decide splits by density or prior strength, not by whether a shared hidden cause is needed.
The mechanism comes from the identifiability literature. Take a lemma $w$. For each occurrence, build three non-overlapping context views $V_1, V_2, V_3$: the left window, the right window, and a sentence-external window. Discretize each view into $m$ codes. If the views are independent once the sense $S$ is fixed, their joint distribution decomposes as
$$P(V_1,V_2,V_3) = \sum_{s=1}^{K} \pi_s \, p_1(\cdot \mid s) \otimes p_2(\cdot \mid s) \otimes p_3(\cdot \mid s),$$
where $\pi_s$ is the frequency of sense $s$, $p_j(\cdot \mid s)$ is the code distribution of view $j$ under that sense, and $K$ is the number of senses. Two facts follow. The rank of this table is $K$. Under mild genericity conditions, the three-way decomposition is unique up to relabeling, so sense assignments are identifiable too, not just their number.
This is the setting of Identification of Latent Confounders via Investigating the Tensor Ranks of the Nonlinear Observations. That paper shows that discrete hidden variables can be identified from continuous nonlinear observations under a "two sufficient measurements" condition. It also shows that a completeness condition makes discretization lossless for rank. We supply the method from this work but apply it to a new object: per-word sense inventories from contextual embeddings. The original work was demonstrated on small social-science tables such as the Bollen industrialization data.
Why Entanglement Is the Real Obstacle
The rank argument breaks in a specific, predictable way, and the attribute-entanglement literature describes it well. Generative models couple attributes that co-occur in training data. Smile and gender shift together, and in text-to-image models "pink" leaks onto a yellow sunflower (Concept Bottleneck Generative Models and A Probabilistic Hard Concept Bottleneck for Steerable Generative Models document the related problem of concept leakage through soft bottlenecks).
In word senses, the entangled nuisance attribute is document topic and register. A monosemous word like "algorithm" appears in machine-learning papers and in news about social media. Both of its context windows then shift together with topic. That inflates the rank of the joint table, which looks exactly like a second sense. We read this as the root cause of the over-splitting that In the LLM era, Word Sense Induction remains unsolved reports. This is a hypothesis, and the program tests it directly.
The pure-child literature tells us how to separate the two hidden causes. Causal Structure Recovery with Latent Variables under Milder Distributional and Graphical Assumptions proves that one pure child per hidden variable can suffice. A pure child is an observed variable caused only by that hidden variable. The same paper shows that "pseudo-pure pairs", children that share a parent but also influence each other, still carry identifying signal.
We map this onto text as follows. Distant document content, with the target sentence removed, serves as a candidate pure child of topic. The local windows are children of both sense and topic. We therefore estimate topic from the distant content, discretize it, and test the rank of the local-view tables conditional on topic. If topic is the only shared nuisance, the conditional rank equals the true sense count.
From Efficient and Trustworthy Causal Discovery with Latent Variables and Complex Relations we take a second ingredient. That algorithm raises an error rather than outputting a graph when purity assumptions fail. Our analog is a goodness-of-fit check on the fitted decomposition. If the residual conditional dependence between views is significant at every $K$ up to a cap, the method abstains and assigns one sense. Given the dominance of monosemous words, abstaining to one sense is the empirically safe default.
Research Program
The procedure has four steps.
- Views. For each lemma, encode every occurrence with a frozen masked-language-model encoder, using the three windows separately. The target token is masked so that its surface form does not leak across views.
- Discretization. Quantize each view with $k$-means into $m$ codes, choosing $m$ larger than any plausible sense count. Do the same for the distant-document topic variable, using $t$ codes.
- Rank selection. For $K = 1, 2, \dots$, fit a nonnegative rank-$K$ decomposition within each topic stratum. Test the residual dependence with a bootstrap likelihood-ratio statistic $\Lambda_K$. Pick the smallest $K$ whose $\Lambda_K$ is not significant, or abstain to $K=1$ if none passes. Because the test's power grows with occurrence count, rare lemmas default to one sense automatically. This is the per-lemma, sample-size-aware behavior that a global concentration parameter cannot provide.
- Assignment. Assign each occurrence to the sense with the highest posterior under the fitted components.
The analysis has three parts. First, a synthetic study generates text-like data from a known sense variable and a known topic variable while varying three things: topic strength, the degree of cross-window dependence (which violates conditional independence), and sample size. This maps where the rank test recovers $K$ and where the abstention correctly fires. Second, we characterize, on real data, which words the method splits and which it abstains on. Third, we probe the main threat to the approach: residual dependence between windows from syntax and collocations. We compare windows separated by gaps of different sizes.
Baselines
The naive reference point is the one-cluster-per-lemma rule. In the LLM era, Word Sense Induction remains unsolved reports 73.6 F-B³ for it on SemCor-WSI. F-B³ is the B-cubed F-measure, which scores each instance by the precision and recall of its assigned cluster. This is the number to beat. Any method that loses to it is over-splitting on net. We will reproduce it on our exact evaluation split before comparing.
The same paper reports 64.9 F-B³ for GPT-4o on SemCor-WSI. We include prompted language models as a reference, not as the main competitor. The file cut off before the score of the paper's strongest encoder-based clustering method: no reported result found on Lacuna for the best BERT-based WSI system on SemCor-WSI. We will rerun that method with the same encoder we use and report its score alongside ours.
The strongest existing competitors are non-parametric clusterers run on the identical encoder outputs:
- Affinity Propagation over instance embeddings.
- DP-means as used in Infinite Mixture Chaining, whose penalty $\lambda$ sweeps from exemplar to prototype models.
- A Dirichlet-process mixture in the spirit of A Bayesian Nonparametric Framework For Learning Disentangled Representations.
Every baseline's global hyperparameter (the clustering threshold, $\lambda$, or the concentration) gets the same tuning budget, selected on the same held-out lemmas we use to set our significance level and $m$. Beating them means higher F-B³ than the best tuned baseline and also higher than 73.6, with a confidence interval from bootstrap over lemmas.
Two ablations isolate the key ingredients:
- Clustering versus identifiability. Fit a Gaussian mixture with BIC model selection per lemma on the same concatenated views. This asks whether the gain comes from counting shared causes or merely from per-lemma model selection.
- Topic conditioning and abstention. Run the rank test without topic conditioning, and separately without abstention. Each should over-split relative to the full method if the entanglement hypothesis is correct.
Why not just…?
Why not just tune the clustering threshold or the Dirichlet concentration on a development set? A single global knob trades over-splitting of frequent monosemous words against under-splitting of polysemous ones. The process prior in Dirichlet models spawns new clusters at a rate tied to sample size, not to evidence of a shared cause. Mosolova et al. evaluate tuned unsupervised systems and still find that none beats one cluster per lemma under natural distributions. Our tuned DP-means and Affinity Propagation baselines test this directly. If a tuned threshold matches us, the identifiability framing adds nothing on this benchmark, and we will say so.
Why not just do per-lemma model selection with BIC on the embeddings? This is the most serious alternative, because it is also per-lemma and sample-size aware. The difference is what counts as structure. BIC rewards any multimodality in the embedding cloud, including topic-driven modes. The rank test counts only dependence that is shared across separated views and survives conditioning on topic. The BIC ablation on identical views is the decisive comparison. If BIC wins, our central hypothesis is wrong.
Why not just ask a large language model how many senses a word has? Mosolova et al. report that GPT-4o and Llama models fall below specialized encoders and hallucinate spurious senses on rare words, scoring 64.9 F-B³ against 73.6 for the trivial baseline. We include the prompting baseline for completeness but expect it to stay below.
Why not just predict polysemy from a dictionary's sense counts? This uses lexical-resource supervision, and with it the task stops being induction. We still run it as an upper-reference baseline: a polysemy classifier trained on WordNet sense counts for lemmas outside the evaluation set, gating whether to cluster at all. It is a real threat on English. The method's argument rests on matching it without supervision. Its value would be clearest for languages and historical periods that have no curated inventory.
Proposed Experiments
The primary benchmark is SemCor-WSI with F-B³, with the 73.6 one-cluster score as the bar. We also report the absolute error in predicted sense count per lemma, split by gold polysemy, so that gains on monosemous words cannot hide losses on polysemous ones. SemEval-2010 and SemEval-2013 WSI serve as secondary, artificially balanced benchmarks. We found no reported result on Lacuna for these in the file, so we will run all baselines ourselves.
The synthetic study is the first decisive check. It should show the conditional rank test recovering the planted $K$ under strong topic effects, where the unconditional test and BIC over-count. As a stretch test of whether identifiable sense counts track meaning change over time, we compare estimated per-period rank changes against SemEval-2020 Task 1 human judgments, using Spearman correlation. No reported value from Infinite Mixture Chaining appears on Lacuna for this comparison, so we rerun it.
Three results would refute the proposal:
- The full method failing to exceed 73.6 F-B³ on SemCor-WSI.
- BIC on identical views matching it.
- Topic conditioning making no difference to over-splitting.
Any one of these would mean that identifiability theory does not fix the specific failure that made sense induction look unsolved.
References
- In the LLM era, Word Sense Induction remains unsolved
- Identification of Latent Confounders via Investigating the Tensor Ranks of the Nonlinear Observations
- Concept Bottleneck Generative Models
- A Probabilistic Hard Concept Bottleneck for Steerable Generative Models
- Causal Structure Recovery with Latent Variables under Milder Distributional and Graphical Assumptions
- Efficient and Trustworthy Causal Discovery with Latent Variables and Complex Relations
- Infinite Mixture Chaining: An Efficiency-Based Framework for the Dynamic Construction of Word Meaning
- A Bayesian Nonparametric Framework For Learning Disentangled Representations