Idioms such as "spill the beans" mean something their words do not add up to. A common way to detect them inside a language model is to ask how far the model's vector for a phrase sits from what its parts would predict. The current best training-free version of this test assumes the parts combine by plain addition and subtraction. Real Transformers do not compose literal phrases additively either, so ordinary phrases also leave a residue, and the residue of an idiom has to be picked out against that background. We propose to learn the model's actual literal composition rule. We do this by linear regression from sub-phrase vectors to phrase vectors, fit on a large pool of unlabeled phrases. A phrase is then called non-compositional when this regression cannot predict it.
The regression has two problems, and the randomized least-squares literature addresses both. First, the pool of training phrases is unlabeled, so it contains some idioms. An ordinary fit absorbs them, and the very phrases we want to find shrink toward zero residual. Second, the regression has thousands of input features and only a few thousand phrases. In this nearly square regime, a single least-squares fit is unstable. Our method fits the map many times on random subsets of phrases. It scores each phrase only with the fits that did not see it, combines those scores robustly, and peels off the worst-predicted phrases one at a time. The result is an unsupervised idiom detector that needs neither curated literal controls nor labels.
Core Claim
Non-compositionality detection can be posed as recovering a small set of outliers in a regression. For a candidate phrase $p$ with Möbius sub-spans $s_1,\dots,s_k$ (for a three-word phrase: the two bigrams and the shared middle word), let $T(\cdot)$ denote a layer's span embedding. We model
$$T(p) = W\,[T(s_1);\dots;T(s_k)] + b + o_p + \epsilon_p,$$
where $W$ and $b$ form a linear composition map shared by all phrases, $\epsilon_p$ is small noise, and $o_p$ is an "idiomatic offset" that is zero for most phrases. Finding idioms means finding the phrases with nonzero $o_p$, which is a sparse support-recovery problem.
The Möbius synergy residual of A Compositional Calculus for Semantic Synergy in Language Model Embeddings is the special case in which $W$ is fixed to the inclusion–exclusion coefficients $(+1,+1,-1)$ and $b=0$. Our first claim is that learning $W$ from the corpus removes systematic non-additivity that literal phrases share, and so separates idioms from literal phrases more sharply.
Our second claim is about how the map is fit. Fitting by random subsampling with out-of-bag scoring and robust aggregation recovers the outlier set better than a single regularized fit when the phrase pool is contaminated. Here "out-of-bag" means scoring each phrase only with the fits trained on subsets that excluded it.
Concretely, we draw $m$ random subsets $S_j$ containing a fraction $n_0/N$ of the $N$ phrases, and fit a ridge map $(W_j,b_j)$ on each. The score of phrase $p$ is
$$r(p) = \mathrm{median}_{j:\,p\notin S_j}\ \big\| T(p) - W_j[T(s_1);\dots;T(s_k)] - b_j \big\|_2 .$$
After scoring, the top-scoring phrase is removed and the fits are recomputed. This is repeated, with majority voting across several subsampling ratios to decide which phrase to remove.
Relation to Prior Work
Three papers motivate the gap. Jansma's Möbius calculus establishes that a residual over the sub-span poset carries idiomatic meaning in middle-to-late layers. Its training-free test reaches a ROC-AUC of 0.90–0.92 on 107 idiom/literal-control pairs. Two of its features are the ones we change: the composition coefficients are fixed, and the method relies on hand-matched literal controls. Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models and Memorization or Reasoning? Exploring the Idiom Understanding of LLMs document that literal word meaning leaks into idiomatic processing. That leakage is the reason a fixed additive model of composition is a poor reference point.
Idiom-Aware Compositional Distributed Semantics trains a separate literal composer and a gated idiom route with supervision. We keep the intuition of "literal composer plus exception" but use no labels, and we read the exception off the composer's failures. In-Context Compositional Learning vis Sparse Coding Transformer uses sparse codes to make composition sharper. It addresses compositional rule transfer only. Our work studies the complementary case, where composition is expected to fail.
The method comes from four sources. From Refined least squares for support recovery we take three things: greedy selection by averaging least-squares fits over random row subsets, majority voting across subset sizes, and the observation that subsampling stabilizes nearly square systems. That paper reports optimal ratios $n_0/N$ typically between 0.6 and 0.8. Reducing Sampling Ratios and Increasing Number of Estimates Improve Bagging in Sparse Regression supports using low sampling ratios with many fits when data are scarce. Recovery Guarantees for Distributed-OMP supports plurality voting over independent fits in noise regimes where single fits fail.
Two other aggregation methods constrain the design. JOBS: Joint-Sparse Optimization from Bootstrap Samples couples the bootstrap fits through a shared sparsity penalty. We test that coupling as a variant: the bags share the outlier support, which is penalized jointly. MMSE Approximation For Sparse Coding Algorithms Using Stochastic Resonance injects noise into inputs to estimate a posterior mean. We do not inject noise into embeddings, because our uncertainty comes from which phrases enter the fit, not from measurement noise on a single phrase.
Research Program
We first build candidate phrase pools from the BNC-derived sentences underlying MAGPIE: A Large Corpus of Potentially Idiomatic Expressions. We extract every verb–determiner–noun, adjective–noun and noun–noun span in context, on the order of $10^4$ phrases. The idiom rate in such a pool is low, which is the sparse-outlier regime we assume.
For each model and layer, we compute the span embeddings $T(p)$ and $T(s_i)$. We then run four estimators on the same pools:
- a single ridge regression with cross-fitted residuals;
- a single trimmed (outlier-dropping) least-squares fit;
- bagged ridge with mean aggregation;
- the full method: subsampled fits, out-of-bag median scoring and greedy RLS-style peeling.
To place the problem in the nearly square regime on purpose, we vary $N$ relative to the feature count $kD$, using principal components to control effective dimension.
We analyze three things. The first is the learned $W$: how far it departs from the Möbius coefficients, and at which layers. The second is the spread of the out-of-bag scores, which we use as a per-phrase confidence. The third is token-level scoring, in which the same map is applied to an expression in each of its contexts. Token-level scoring addresses potentially idiomatic expressions whose reading depends on the sentence. For example, "break the ice" can be literal in one sentence and figurative in another.
Baselines
Naive reference. Naive unigram subtraction, the phrase vector minus the sum of its word vectors, is the floor that the Möbius paper already shows to be weak.
Strongest existing training-free method. This is Möbius synergy from A Compositional Calculus for Semantic Synergy in Language Model Embeddings, which reports a ROC-AUC of 0.90–0.92 and Cohen's d of about 1.80 on its 107-pair benchmark. We recompute it with the same models, layers and span-pooling choices we use. Beating it means exceeding its AUC on the 107-pair set and on the 836-pair expansion, at matched layers. We will also report performance when the matched controls are replaced by our unlabeled pool, the setting we target.
Single cross-validated ridge. This is the decisive ablation for the learned map. It receives the same pools, the same feature construction, and a $\lambda$ grid of equal size to our $n_0/N$ grid.
Single trimmed least squares. This is the decisive ablation for randomization. Its trimming fraction and iteration count are tuned on the same budget. It shows whether robustness alone, without subsampling, explains any gain.
Supervised reference. For token-level disambiguation we report the SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding winners: Subtask A macro F1 of 0.8895 zero-shot and 0.9385 one-shot. These systems are trained classifiers. We do not claim to beat them with an unsupervised score. We do claim that adding our score as a feature to a fine-tuned classifier of the same size raises zero-shot F1.
Why not just…?
Why not just fit one ridge regression with a tuned penalty? In a clean pool, subsampled least squares acts much like ridge, so this is a real threat. Our expected advantage appears only under contamination: a single fit spends capacity predicting the idioms it was trained on. We settle this with injection experiments. We plant known idioms from EPIE Dataset: A Corpus For Possible Idiomatic Expressions into literal pools at 1%, 5% and 20% contamination and plot recovery precision. If ridge tracks our method at every rate, we will report that randomization is unnecessary and keep the learned map as the contribution.
Why not just use Möbius synergy as is? It needs no training and is strong. Its fixed coefficients, however, treat all literal non-additivity as signal, and the published AUC leaves room for error. If the learned $W$ stays close to $(+1,+1,-1)$ at the informative layers and gives no AUC gain, the first claim fails. That outcome would itself confirm the Möbius calculus.
Why not just use one robust regression such as trimmed least squares? Trimming handles contamination but must judge outliers with a fit that the outliers have already influenced. RLS-style averaging over fits that exclude each phrase avoids this. The ablation that compares trimmed least squares against the full method on identical pools answers the question directly.
Why not just fine-tune a classifier? Classifiers need labels in the target language and inherit literal leakage on unseen expressions, as Memorization or Reasoning? Exploring the Idiom Understanding of LLMs shows for lower-resource languages. Our method needs only raw text, so its natural test is the Portuguese and Galician zero-shot portion of SemEval-2022 Task 2. On that portion we compare F1 against the fine-tuned classifier with and without our feature.
Proposed Experiments
At the type level, we evaluate on the Möbius 107-pair, 836-pair and 96-name benchmarks by ROC-AUC per layer. We also evaluate on MAGPIE types by average precision for idiomatic types against the unlabeled pool. At the token level, we use MAGPIE and SemEval-2022 Subtask A (macro F1, zero-shot and one-shot).
Four checks are decisive. The first is AUC above 0.92 on the 107-pair set without matched controls. The second is a contamination curve on which our method degrades more slowly than both ridge and trimmed least squares. The third is an interior optimum in $n_0/N$ consistent with the 0.6–0.8 range reported by RLS. If the best ratio is 1, subsampling has added nothing. The fourth is out-of-bag score variance that predicts errors, measured as higher variance on misclassified phrases.
We will also report runtime relative to a single fit, since the ensemble scales linearly with $m$.
References
- A Compositional Calculus for Semantic Synergy in Language Model Embeddings
- Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models
- Memorization or Reasoning? Exploring the Idiom Understanding of LLMs
- Idiom-Aware Compositional Distributed Semantics
- In-Context Compositional Learning vis Sparse Coding Transformer
- Refined least squares for support recovery
- Reducing Sampling Ratios and Increasing Number of Estimates Improve Bagging in Sparse Regression
- Recovery Guarantees for Distributed-OMP
- JOBS: Joint-Sparse Optimization from Bootstrap Samples
- MMSE Approximation For Sparse Coding Algorithms Using Stochastic Resonance
- MAGPIE: A Large Corpus of Potentially Idiomatic Expressions
- SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding
- EPIE Dataset: A Corpus for Possible Idiomatic Expressions