Is Idiomatic Sentiment Low-Rank? Principal Directions of the Compositionality Residual

When a phrase like "to die for" or "break a leg" means something its words do not, a language model has to add meaning that is not in the parts. This proposal measures that added meaning directly. For each phrase, we take its representation in context and subtract a "literal" reference built from its words in isolation. We then run principal component analysis (PCA, the classic method that finds the few orthogonal directions along which a cloud of vectors varies most) on these leftovers across thousands of phrases. The claim is that the leftover, which we call the compositionality residual, is low-rank. A handful of shared directions carry most of it, and one of them lines up with sentiment reversal. If that holds, two things follow without any phrase-level sentiment labels. First, we can score how non-compositional a phrase occurrence is. Second, we can correct a word-lexicon sentiment estimate that is known to fail on idioms.

The motivation is a sharp negative result. Confirming the Non-compositionality of Idioms for Sentiment Analysis summed dictionary pleasantness scores over idiom words and compared them with crowd labels from the SLIDE idiom lexicon. The correlation was essentially zero, and for positive idioms it was slightly inverted (Spearman ρ = −0.144). Separately, the PCA literature shows that transformer representations are highly redundant. Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval finds that about 256 of 768 dimensions carry ~90% of the variance of dense retrieval vectors. La RoSA shows that PCA rotations concentrate the information in LLM activations. Nobody has asked whether the specific part of a representation that composition fails to explain is itself concentrated in a few directions.

Core Claim

Write $h(p, c)$ for the hidden state of phrase $p$ occurring in sentence $c$, mean-pooled over the phrase's tokens at a chosen layer. Write $\bar{h}(w)$ for the average hidden state of word $w$ over many random corpus occurrences. The residual is

$$r(p,c) = h(p,c) - \frac{1}{|p|}\sum_{w \in p} \bar{h}(w).$$

We fit PCA on residuals pooled from unlabeled occurrences of both idioms and ordinary n-grams, after removing the mean and the top few generic directions. That removal step is the post-processing of Effective Dimensionality Reduction for Word Embeddings, which found that such dominant directions encode frequency rather than meaning. We then measure a sentiment axis $s$ in word space from a unigram lexicon, as the difference between the mean positive-word and mean negative-word vectors. The predicted phrase sentiment is

$$\hat{y}(p,c) = \ell(p) + \alpha \, s^\top P_k \, r(p,c),$$

where $\ell(p)$ is the literal lexicon score, $P_k$ projects onto the top $k$ residual principal components, and $\alpha$ is either fixed at 1 (zero labels) or fit on a few dozen labeled phrases.

The claim is falsifiable in three ways. The first is that residual variance for idioms should concentrate in far fewer components than residual variance for compositional phrases. The second is that $\|P_k r\|$ should separate idiomatic from literal uses of the same expression. The third is that the correction term should raise the correlation with SLIDE labels from near zero to clearly positive. The strongest version of the third test is on "trap" idioms, whose literal words have the opposite polarity to the idiom.

Relation to Prior Work

Two papers are closest, and both supply parts of the method rather than the question. Ultradense Word Embeddings by Orthogonal Transformation learns a supervised rotation that squeezes sentiment into 1–4 dimensions. It reaches Kendall τ = 0.654 on SemEval-2015 lexicon creation, but it works only on single words and needs polar seed pairs. We borrow its idea that sentiment is a low-dimensional subspace, but we apply unsupervised PCA to residuals of phrases rather than to word vectors. Learning Sentiment Composition from Sentiment Lexicons models reversers and dominators in bigrams, reaching 0.76 precision. It uses singular value decomposition only as a compression step for co-occurrence vectors and never inspects the components. We make the components themselves the object of study, and we extend the analysis to three-to-five-word idioms in context.

The idiom-modeling line constrains the design. IDEM finds that RoBERTa needs the idiom's dictionary definition to reach 58.73% weighted F1 on emotion recognition. Enhancing Language Models with Idiomatic Reasoning builds Idapter, a supervised adapter with non-compositionality scaling. Both depend on curated definitions or supervised tuning; our method needs neither.

MAGPIE warns that most potentially idiomatic expressions lean heavily toward one sense: only 4.6% are balanced between literal and idiomatic use. This means per-type sense frequency is a strong confound that our evaluation must control. SemEval-2016 Task 7 provides modifier phrases such as "not that good". These are non-additive but largely rule-governed, and they serve as an intermediate control group.

Research Program

We extract residuals from RoBERTa-large and from one open decoder model, across all layers. The contexts come from three sources: the 9,685 IDEM sentences covering 2,322 SLIDE idioms, the literal and idiomatic instances in MAGPIE, and SemEval-2016 English phrases embedded in retrieved corpus sentences. As a compositional contrast set, we add frequent ordinary bigrams and trigrams in random corpus sentences.

The analysis has four stages, run separately for idioms, modifier phrases, and ordinary n-grams.

The first stage asks whether the residual is low-rank. We compare scree curves (the variance explained by each successive principal component) and effective rank.

The second stage asks whether the dominant residual directions are shared across idioms. We fit PCA on half the idiom types and measure explained variance on the other half.

The third stage checks alignment. We measure how strongly the sentiment axis $s$ projects into $P_k$, compared with its projection into random $k$-dimensional subspaces of the same model.

The fourth stage uses the residual subspace in two applications. One is unsupervised detection of idiomatic versus literal use. The other is sentiment correction via $\hat{y}$.

The layer and $k$ are chosen on a development split of idiom types. PCA fitting never sees sentiment labels.

Baselines

The naive reference point is the component-wise lexicon score $\ell(p)$ alone. We reimplement it in the style of Hwang and Hidey. The published anchor is Spearman ρ = −0.144 on positive SLIDE idioms, with no significant correlation on neutral or negative idioms (Confirming the Non-compositionality of Idioms for Sentiment Analysis). Beating it means a significantly positive ρ on all three classes, under the same SLIDE Positive Percent Index labels.

The strongest competitor for sentiment is a linear probe (a logistic or ridge regressor) trained on the full contextual phrase embedding $h(p,c)$. It is strongest because it sees everything the residual method sees, plus the literal content. It gets the same labeled budget as our few-label variant (25, 100, and 400 idiom types). It is evaluated on held-out idiom types, with an identical regularization search. We count the proposal as beating it if the residual method matches or exceeds it on the trap subset with fewer labels, or with zero labels.

For downstream reference points, we report the published IDEM figures: 58.73% weighted F1 for idiom-aware RoBERTa and 61.05% for zero-shot GPT-4 (IDEM). We rerun an idiom-ignorant RoBERTa on the same splits. For SemEval-2016 General English, the published best is Kendall τ = 0.70, from team ECNU (SemEval-2016 Task 7). For MAGPIE literal-versus-idiomatic detection, no reported result was found on Lacuna. We therefore train a supervised RoBERTa classifier ourselves as the upper reference and a majority-sense-per-type classifier as the confound baseline.

Three ablations isolate the key ingredients. The first is residual norm without PCA ($\|r\|$), which tests whether the principal-direction structure matters beyond the size of the deviation. The second is PCA on raw $h$ instead of on $r$, which tests whether subtracting the literal reference matters. The third is residual PCA without removing the generic top directions, which tests the post-processing step. As a further supervised comparison, we apply a DENSIFIER-style supervised rotation to phrase embeddings under the same label budget.

Why not just…?

Why not just train a linear probe on the full contextual embedding? This is the real threat. On in-distribution idiom types it may match our method. The residual approach earns its keep only where the probe is likely to latch onto literal word polarity. That happens on trap idioms like "to die for" and on idiom types unseen during training. The held-out-type, trap-subset comparison at small label budgets settles this. If the full probe wins there too, we report that the sentiment claim fails. The contribution then reduces to the unsupervised geometric finding and the idiomaticity detector.

Why not just prompt a large language model? Zero-shot GPT-4 already reaches 61.05% weighted F1 on IDEM (IDEM), and we do not expect to beat it on raw accuracy. The proposal is a claim about where non-compositional meaning lives inside an encoder, and it yields a cheap, inspectable score. Prompting gives neither. We therefore report the GPT-4 figure as a ceiling rather than as a competitor.

Why not just use raw PCA on phrase embeddings, as in the retrieval compression work? Simple and Effective Unsupervised Redundancy Elimination shows that raw PCA preserves overall semantic geometry. The literal content, however, dominates that variance, and literal content is exactly what misleads on idioms. The raw-$h$ PCA ablation tests this directly.

Why not just look up idioms in a dictionary? MAGPIE's literal instances show that a lookup cannot tell "spilled the beans on the floor" from confessing (MAGPIE). Our literal-versus-idiomatic experiment on the same expression types tests exactly this contextual distinction.

Decisive Checks

Three results would confirm the idea.

First, the idiom residuals should need markedly fewer components than ordinary n-gram residuals to reach the same explained variance. This structure should transfer to held-out idiom types. It should also vanish when phrase–context pairings are shuffled.

Second, projecting residuals onto the top components should separate literal from idiomatic MAGPIE instances within the same expression type. The within-type evaluation rules out the per-type sense-skew confound. This separation should also beat the plain residual norm.

Third, the zero-label correction $\hat{y}$ should turn the near-zero SLIDE correlation positive on trap idioms. The SemEval-2016 modifier phrases should fall between idioms and ordinary n-grams in residual rank. That ordering would show the measure tracks degree of non-compositionality rather than phrase length or rarity.

If the first check fails, the idea is wrong. If only the third fails against the full probe, the geometric claim stands but the practical sentiment claim does not.