Retrieved Usage Evidence as a Language-Relative Measure of Metaphor Novelty

Language models handle worn-out metaphors like "spending time" well and fail on fresh ones like "unzipping a shadow." This pattern is called conventionality bias. The usual explanation is word frequency, and the usual fix is a better detector. This note starts from a different observation. Whether a metaphor is conventional is not a property of the expression. It depends on the language and the reader. "Building a project" is routine in English and Spanish. An English metaphor that German translators literalize about half the time is probably not routine in German. We propose to measure conventionality directly: retrieve passages from a text collection in each language, and count how often the same figurative mapping (the same borrowing of meaning from one domain, such as clothing, into another, such as light) actually appears there. We then test whether this count explains three things that are currently studied separately. The first is when models fail on figurative language. The second is when human annotators disagree about novelty. The third is when translation drops a metaphor.

The practical payoff is a retrieval-augmented generation system for figurative language. Retrieval-augmented generation, abbreviated RAG, means a model that looks up passages from an external collection before answering. In our version the system does not pivot to English by default, as current cross-lingual question-answering systems do. It routes the lookup to whichever language treats the metaphor as ordinary, and shows the model real usage examples from that language. If the idea is right, retrieval should help most on exactly the items that are novel in the query language but conventional somewhere else. That is a sharp, falsifiable prediction.

Core Claim

Our claim is that an estimate of how conventional a figurative mapping is in language $L$, computed from retrieved and verified usage examples, predicts three things better than word frequency does. It should predict model errors on figurative inference, the size and direction of disagreement among human novelty ratings, and whether a metaphor survives translation. The claim has two parts. First, the same estimate computed per language accounts for cross-lingual differences in all three. Second, feeding the retrieved usage examples to a generator improves figurative inference, with gains concentrated where query-language and other-language conventionality diverge.

The estimate is simple. For an expression $x$ in context, we generate a short description of its figurative mapping. We retrieve the top $k$ passages from a collection in language $L$ using a multilingual dense retriever, which matches text by meaning-level vector similarity rather than shared words. A verifier model then decides, passage by passage, whether each one uses the same mapping figuratively. The attestation score is

$$A_L(x) = \log\left(1 + \frac{h_L(x)}{k}\cdot \frac{N_{ref}}{N_L}\right),$$

where $h_L(x)$ is the number of verified hits among the $k$ retrieved passages, $N_L$ is the size of the language-$L$ collection, and $N_{ref}$ is a fixed reference size. The size ratio makes scores comparable across collections of unequal size. We use this one equation throughout. Novelty in language $L$ is predicted to fall as $A_L$ rises.

Why This Is Not Already Done

Four lines of prior work each cover part of this ground.

The first line motivates the proposal by showing that current novelty measures confound rarity with creativity. When is a Metaphor Actually Novel? found that crowdsourced novelty scores correlate with word frequency at −0.612. It then replaced crowd judgments with an offline dictionary check: a metaphor counts as novel when its contextual sense is absent from the Longman dictionary. Weeding out Conventionalized Metaphors used Best-Worst Scaling, a rating method where annotators pick the most and least novel item from small sets, to produce continuous novelty scores for 15,180 English metaphors. It found a frequency–novelty correlation of ρ = −0.60. Both studies are English-only. Both treat conventionality as a fixed label rather than something measured live from corpus evidence. Our attestation score is a corpus-grounded, per-language replacement for the dictionary check, and it can be computed for languages that lack a comparable dictionary.

The second line constrains the proposal. It shows the gap is not simply one of detection. An analysis of language models for metaphor recognition showed that BERT falls from above 70% accuracy on conventional metaphors to below 60% on novel ones. It also showed that performance tracks whether a word type or its synonyms were seen in training. That is a parametric version of our attestation idea, but it is measured against a training set rather than a language. Figurative Language in Recognizing Textual Entailment showed that textual-entailment models stay near chance on some figurative categories.

The third line documents cross-lingual drift without explaining it. VOLIMET found that French preserves English verb-object metaphors 63% of the time and German only about 50%, with figurativeness agreement around κ ≈ 0.41. Meta4XNLI calls translation a "metaphor killer and creator." It shows that metaphor degrades entailment accuracy across English and Spanish, but it keeps only curated gold labels. We ask whether per-language attestation predicts which metaphors translation kills. We also ask whether it predicts where English and Spanish readers disagree.

The fourth line supplies the retrieval machinery and the assumption we reject. XOR QA and XRAG retrieve across languages, mostly pivoting to English, for literal factual questions. XRAG adds a warning we must respect: models sometimes answer in the language of the retrieved passage rather than the language of the question.

The closest work on soft labels, Prompting Metaphoricity, uses a large language model to produce graded metaphoricity labels for Spanish tweets. Those labels come without external evidence. We ground them.

Research Program

The work builds one artifact and runs three analyses and one intervention on it.

The artifact is the per-language attestation estimator, built over Wikipedia and general web text in English, Spanish, German, and French. The components are a mapping-description generator, a multilingual dense retriever, and a verifier. We will release the verifier's decisions so its error rate can be audited on a hand-checked sample. Verifier quality is the main technical risk. If the verifier cannot tell figurative from literal use of the same words, $A_L$ collapses into a smoothed frequency count, so the ablations below are designed to catch exactly that.

The first analysis asks whether attestation measures conventionality or merely rarity. The data are the English novelty scores and the dictionary-based novel set. The second analysis asks whether per-language attestation predicts VOLIMET's metaphor preservation in German and French translations. The third analysis addresses annotation subjectivity. We collect new Best-Worst novelty ratings from separate groups of native English and native Spanish speakers on about 1,000 parallel metaphor pairs from Meta4XNLI. We keep every individual rating, not a single aggregated gold label. We then test two predictions. The between-group rating gap should track $A_{en}(x) - A_{es}(x)$. Within-group rating spread should be largest where attestation is intermediate, that is, where conventionalization is partial.

The intervention is attestation-routed RAG. For a figurative inference item, the system computes $A_L$ across languages. It retrieves verified usage examples from the language with the highest attestation and places them in the generator's prompt. It then instructs the generator to answer in the query language. We check that last step with XRAG's response-language measure.

Baselines

The naive reference point is the same generator with no retrieval. On each benchmark we compare it against the fine-tuned encoder results reported in the file.

The strongest existing predictor of novelty is word frequency. Weeding out Conventionalized Metaphors reports ρ = −0.60 against its Best-Worst scores and a BiLSTM regression error (mean absolute error) of 0.163 on the −1 to 1 scale. We compute frequency from the same collections used for retrieval, so neither side has a data advantage. We fit both predictors with identical regression heads and identical cross-validation folds. Beating this baseline means a significant partial correlation between attestation and novelty after controlling for frequency, plus a lower mean absolute error than both frequency alone and 0.163. We also require the advantage to be larger on the dictionary-based target than on the crowdsourced one. Frequency's strength on crowd scores partly reflects annotator bias, and a better measure of conventionality should pull ahead where that bias is removed.

The strongest detector on the dictionary-based novel set is DeepMet, which When is a Metaphor Actually Novel? reports at 63.10% recall. We use this number as a reference for whether routed retrieval lifts novel-item performance.

For translation preservation, the baseline is English-side novelty plus the translation model's own probability of a literal rendering.

For downstream inference, the strongest competitor is English-pivot retrieval, in the style of XOR QA and XRAG, using the same retriever, generator, number of passages, and prompt length. On XRAG, the best model, GPT-4o, reaches 62% in English against 85% for humans. For metaphor-affected entailment pairs, however, no reported result was found on Lacuna for Meta4XNLI accuracy, and none for the metaphor category of the figurative entailment benchmark. We therefore rerun every system ourselves.

The key ablation replaces verified same-mapping usage examples with passages that share the target words but were rejected by the verifier. This gives the same retrieval budget without the mapping evidence. A second ablation retrieves dictionary or Wiktionary definitions instead of usage examples. Beating the English-pivot baseline means a significant accuracy gain on items where the Spanish and English attestation scores diverge, with no gain required on items where they agree.

Why not just…?

Why not just use word frequency, which already correlates at −0.60 with novelty? This is the real threat. Attestation computed from a corpus could turn out to be frequency in disguise. Two things in the file suggest it is not. Frequency's correlation with crowd scores partly reflects annotators mistaking rare words for novel metaphors. And An analysis of language models for metaphor recognition shows that sense-level exposure, through synonyms and morphological variants, matters beyond raw counts. We settle the question empirically with the partial-correlation test and with a stratified analysis. One stratum is common words used in novel senses. The other is rare words used in their conventional senses. If attestation adds nothing on these strata, we will report that the core claim fails.

Why not just ask a large model to judge novelty or interpret the metaphor directly? Meta4XNLI finds decoder models trail fine-tuned encoders at metaphor detection. XRAG finds that even GPT-4o falls well short of humans on cross-lingual synthesis. Neither result directly tests figurative interpretation by today's strongest generators, so the no-retrieval condition with the same generator is a genuine competitor. It is also the exact comparison our downstream experiment makes. If the no-retrieval model matches routed RAG on the items where attestation diverges across languages, the intervention is unnecessary. The measurement results would still stand.

Why not just look the sense up in a dictionary, as the dictionary-based novelty standard does? Dictionaries record which senses exist, not how common they are. Coverage is also uneven across languages. The definition-retrieval ablation tests directly whether usage examples beat definitions for downstream inference.

Why not just translate everything into English and use English retrieval? VOLIMET and Meta4XNLI show that translation drops and invents metaphors. More importantly for our claim, English is by construction the wrong place to look when a mapping is conventional in Spanish but novel in English. The English-pivot baseline measures this directly.

Decisive Checks

Three results would decide the proposal.

The first test is on the English novelty scores and the dictionary-based novel set. Attestation must retain predictive power after controlling for frequency, and its advantage over frequency must be larger on the dictionary-based target than on the crowdsourced one. If it fails, the estimator is a frequency proxy and the program stops at a negative result.

The second test is on VOLIMET. Low German and French attestation should predict literalization beyond what English-side novelty predicts. If per-language scores add nothing, the "language-relative" part of the claim is false.

The third test is on our new English and Spanish ratings and on Meta4XNLI. The between-group rating gap should track the attestation difference. Routed retrieval should beat English-pivot retrieval specifically on items where the two languages' attestation diverges. The verifier-rejected ablation should erase that gain, and response-language correctness should not fall below the no-retrieval condition.

A pattern where retrieval helps uniformly across all items, rather than where attestation diverges, would mean the benefit comes from generic added context. We would report it that way.