Hide the Idiom: Small-Sample Literal–Figurative Disambiguation by Reading Only the Context

Many phrases can be meant two ways. "Spill the beans" can describe a chef knocking over a pot or a politician leaking a secret, and the words are identical in both cases. Deciding which reading a particular sentence intends is called token-level literal–figurative disambiguation. Current models handle it badly in exactly the setting that matters in practice: labeled examples are scarce, literal uses are rare, and the phrase in question was never seen in training. This note proposes a simple change to what the classifier is allowed to look at. The model never sees the phrase itself. It sees the surrounding sentence with the phrase masked out, plus two short written descriptions of what the phrase would mean literally and figuratively. It is trained on a few dozen examples to judge which description the masked context fits better. Because the decision rule is shared across all phrases and the phrase-specific knowledge comes from text rather than labels, a small model should be able to learn it from very little data.

The bet rests on one diagnosis. Small models fail because they look at the phrase and retrieve its usual, figurative meaning before they ever weigh the context, not because the context is uninformative. Hiding the phrase removes that default, and with it the memorized "this phrase is usually figurative" shortcut that skewed training data otherwise teaches. If the diagnosis is right, a sub-1B model trained on fewer than 100 labels should approach or exceed the controlled-benchmark consistency of 70B-parameter prompted models.

Core Claim

Under small-sample constraints, the obstacle to literal–figurative disambiguation is the phrase's own prior, and removing the phrase from the classifier's input is a structural constraint that pays for itself in sample efficiency. Three findings motivate this. First, Rolling the DICE on Idiomaticity built a benchmark where literal and figurative sentences share the same surface syntax. It found a strong figurative bias, near-zero strict consistency for models under 13B parameters, and lower consistency after one-shot prompting: Gemma 2 9B fell from 28.61% to 15.67%, and Mistral 7B from 12.69% to 2.49%. Strict consistency means getting every literal and every figurative sentence for a given idiom right. Second, Tug-of-war between idiom's figurative and literal meanings in LLMs used causal tracing to show that early-layer feed-forward blocks retrieve a figurative "macro" meaning from the phrase's tokens. The literal reading wins only when specific attention heads actively suppress that retrieval, so figurative is the default track. Third, MAGPIE shows that only 4.6% of idiom types are roughly balanced between senses. A classifier trained on few natural examples can therefore score well by memorizing each phrase's majority sense. All three point the same way: the phrase is the source of the shortcut, and in the controlled contrast DICE sets up, the phrase carries no discriminative signal at all.

The intuition is old. Unsupervised Recognition of Literal and Non-Literal Use of Idiomatic Expressions classified usage by whether the phrase's words "cohere" with the surrounding words: literal uses fit the topic of the sentence, while figurative constituents stick out. Their web co-occurrence measure barely beat the majority class (79.61% vs 78.25% accuracy on the Reuters idiom set). The proposal keeps their question, "does this context fit the literal reading?", but replaces hand-built co-occurrence statistics with a learned comparison between context and description. It also asks the question symmetrically for both readings.

Research Program

The model is a cross-encoder, meaning one encoder that reads two texts jointly and outputs a match score. One text is the sentence with the target expression replaced by a single mask token. The other is a sense description. Each idiom gets two descriptions, written once per idiom and never per sentence. The figurative description is a dictionary gloss, the same kind of resource Unified Representation for Non-compositional and Compositional Expressions aligns to during training. The literal description is a compositional paraphrase built from the constituent words' plain meanings (for example, "causing dried beans to fall out of a container"). We generate it by one fixed LLM pass and spot-check it, and we also test a fully templated variant that needs no generation.

Let $s(c, d)$ be the encoder's score for masked context $c$ and description $d$. Let $d^+$ be the correct description and $d^-$ the other one. Training uses a margin loss.

$$ \mathcal{L} = \max\big(0,\ m - s(c, d^+) + s(c, d^-)\big) + \lambda \max\big(0,\ m - s(c_L, d_L) - s(c_F, d_F) + s(c_L, d_F) + s(c_F, d_L)\big) $$

Here $m$ is a fixed margin. The second term applies only when the labeled set contains a minimal pair, meaning a literal context $c_L$ and a figurative context $c_F$ for the same idiom, with literal and figurative descriptions $d_L$ and $d_F$. It rewards getting the pair right jointly, which is what strict consistency measures. The weight $\lambda$ is fixed in advance, not tuned on test idioms. The pairing borrows the anchor–positive–negative idea from Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Loss. That paper learns representations of idioms that have already been identified, scores them against paraphrases, and replaces each idiom with a single surrogate token. Our model does the opposite: it removes the idiom and classifies the occurrence from context.

The small-sample direction plays two roles. As a regime, all training uses $N \in \{0, 16, 64, 256\}$ labeled instances in total, and the training idioms are disjoint from the test idioms. As a lesson in design, work such as MiME, which builds a known clinical hierarchy into the embedding, and the echocardiography segmentation pipeline, which pairs a shape prior with a deformable contour, succeeded with fewer than 50 samples by fixing structure in advance so that little is left to learn. GraphCare similarly substituted external structured knowledge for samples. It reports that 36 patients matched baselines trained on 720. Here the fixed structure is the input format, a masked context plus two descriptions, and the external knowledge is the description text. The learned part is a phrase-agnostic notion of fit between context and meaning.

The analysis has three parts. We measure figurative bias as the share of literal sentences labeled figurative. We measure shortcut reliance as the correlation, across test idioms, between a model's predictions and that idiom's MAGPIE majority sense. We also run a cost check: does masking lose real cues on natural text? Literal uses of idioms are often more syntactically flexible, so form carries information in MAGPIE even though DICE neutralizes it. For that reason we test a variant that adds only coarse form features of the hidden phrase, such as inflection, inserted modifiers, and passivization, without its lexical identity.

Baselines

The naive reference points are the majority-sense classifier and the reported defaults. On the Reuters idiom set, the all-idiomatic baseline reaches 78.25% accuracy with 0% literal recall, and the unsupervised cohesion graph reaches 79.61% accuracy with 67.87% literal recall (Sporleder and Li). On DICE, models under 13B score near zero strict consistency, with Llama 3 8B at 0.00% (DICE). Beating these is necessary but uninformative.

The strongest existing method on the decisive benchmark is prompting a large LLM. On DICE, Llama 3.1 405B reaches 60.36% strict consistency, Llama 3 70B 57.55%, and GPT-4o 48.59% (DICE). We cite these reported numbers rather than re-running them. We re-run open 7–9B and 70B models zero-shot, and few-shot with our same $N$ labeled examples as demonstrations, so that the data are matched. Beating it means our sub-1B model exceeds 57.55% strict consistency on DICE with $N \le 256$ labels. Matching 60.36% at $N = 64$ would be the headline result.

The strongest fine-tuned competitor is a standard encoder classifier trained with cross-entropy on the full sentence. It uses the same backbone, the same $N$ examples, and the same tuning budget of a fixed small grid over learning rate and epochs, chosen on held-out training idioms only. We also run the encoder with descriptions but without masking, so that the comparison isolates exactly the masking ingredient. A non-parametric baseline in the style of Non-Parametric Few-Shot Learning for Word Sense Disambiguation assigns each occurrence to the nearest gloss in embedding space, and we run it with and without masking. PIER+ (Zeng and Bhat) serves as a fully supervised ceiling: it reports up to 91.07% sense-classification accuracy on MAGPIE with dense supervision. We found no reported result on Lacuna for PIER+ or any other trained method in a few-shot or idiom-disjoint MAGPIE setting, so we will produce those numbers ourselves.

The ablations remove one ingredient at a time: masking, the pair-consistency term, the literal description (scoring against the figurative gloss only), and LLM-generated literal descriptions (replaced by templates).

Why not just…?

Why not just fine-tune a standard classifier on the few labels? This is the main threat on natural, in-domain text. If test idioms overlap training idioms, memorizing each phrase's majority sense is a strong strategy given MAGPIE's skew, and an unmasked classifier may win there. Our claim is narrower: on unseen idioms and on syntax-controlled pairs, that memorization becomes a liability. The idiom-disjoint MAGPIE split and DICE settle the question. We will report the in-domain overlap condition as well, even if it favors the baseline.

Why not just use an off-the-shelf zero-shot entailment model with the two descriptions as hypotheses? This is close to our $N = 0$ condition and is a real competitor. If zero-shot entailment with masking matches our trained model, then the contribution reduces to "mask the phrase." That is still a useful and testable finding, and we would say so. The masked-versus-unmasked comparison at $N = 0$ directly measures whether the phrase prior is the bottleneck. The comparison between $N = 0$ and $N = 64$ measures whether the few labels add anything.

Why not just prompt a larger LLM? The reported ceiling is 60.36% strict consistency at 405B parameters, and one-shot demonstrations hurt mid-sized models (DICE). That makes "add the few labels to the prompt" an unreliable way to use scarce data. Our matched-data prompting runs test this directly at 7–9B and 70B.

Why not just generate many synthetic literal sentences and fine-tune? DICE's literal contexts were themselves GPT-4-generated, so synthetic training data risks teaching the generator's style rather than literalness. Work on limits of synthetic data in data-scarce domains and the diminishing returns reported for augmentation (Jia et al.) add caution. We include one augmentation baseline: the unmasked classifier plus LLM-generated literal sentences up to twice $N$. If it matches our method on Reuters, which uses non-synthetic text, the augmentation route is a viable alternative.

Proposed Experiments

Training draws $N$ labeled instances from MAGPIE idioms that are disjoint from DICE's 402 idioms and from the 17 Reuters idioms. Because DICE was built partly from MAGPIE, this disjointness is mandatory. Each $N$ is run with five random draws, and we report the mean and spread. The backbone is a sub-1B encoder of the size He et al. used, such as xlm-roberta-large.

Evaluation covers four datasets. On DICE we report strict consistency, accuracy, and macro-F1. On a MAGPIE test split with held-out idioms we report accuracy and literal F1. On the Reuters idiom set we report accuracy and literal recall. As a transfer probe, we use the location-metonymy corpus from Metonymy Resolution as a Classification Task. There the literal baseline is 79.7% and a feature-based decision list reaches 84.3%, but its literal-versus-non-literal recall is only 33.9%. This corpus tests whether a phrase-agnostic notion of fit extends beyond idioms when country names are masked and paired with "the place" and "the people or government" descriptions.

The decisive checks are four.

First, at $N = 64$, masked training should exceed the unmasked classifier on DICE strict consistency by a wide margin. If masking does not help on DICE, where the phrase carries no signal, the diagnosis is wrong.

Second, the figurative-bias rate should drop under masking, and the correlation between predictions and each test idiom's majority sense should fall toward zero. If accuracy rises but this correlation stays high, the gain comes from something other than removing the shortcut.

Third, on idiom-disjoint MAGPIE, masked training should beat the unmasked classifier at small $N$. The gap should narrow as $N$ grows, because larger label sets let the unmasked model learn context use despite the shortcut. The form-feature variant shows how much of any remaining gap is syntactic flexibility rather than lexical identity.

Fourth, the result we aim for: our sub-1B model at $N \le 256$ crosses Llama 3 70B's reported 57.55% strict consistency on DICE. If it lands between the matched-data 7–9B prompting runs and 57.55%, the method still beats the cheap alternatives, but it does not close the gap to scale.