Most metaphors work by taking a word with a concrete, physical meaning and using it for something abstract. To "digest" information, for example, is to use a stomach verb for a mental act. The best metaphor detectors encode this intuition with hand-built inputs. They look up a fixed concreteness score for each word from a human-rated dictionary, add part-of-speech tags, and sometimes run a separate parser to produce a syntax tree. In computer vision and medical imaging, a long line of work showed that a network trained end to end on minimally processed input can learn its own features, and that those features can then be checked against what experts already know. We propose the same move for metaphor detection. Instead of feeding the detector a fixed concreteness score per word, we train it to estimate how concrete each word is in its particular sentence. The human dictionary is used only as a weak training signal. The detector then reads off the gap between a word's usual concreteness and its concreteness in this sentence.
The opportunity matters for two reasons. First, the field has named "concreteness in context" as an open problem: type-level norms give "digest" the same score in "digest the meal" and "digest the report". Second, upstream hand-engineered steps break down in exactly the genres where detectors fail. Conversation and fiction have fragmented syntax that parsers and taggers handle poorly. Imaging research showed the same failure pattern, in which strong learned components were undermined by brittle preprocessing. If a learned, context-sensitive concreteness signal can replace dictionary lookups and tagging at test time, we expect the largest gains on the hardest genres. We also expect to be able to verify that the learned signal tracks the source and target words of the metaphor, rather than serving as an uninterpretable shortcut.
Core Claim
A metaphor detector that learns a per-token concreteness estimate will match or beat feature-engineered systems on the VU Amsterdam corpus, and will degrade less on conversation and fiction, without using concreteness norms, part-of-speech tags or parses at inference. The estimate is supervised only by requiring that its average over a word's occurrences agree with that word's dictionary rating. The detector's evidence is the difference between the word's predicted out-of-context concreteness and its in-context estimate.
What is new is the role the norms play. In Metaphor Detection using Context and Concreteness, static Word2Vec vectors, contextual BERT vectors and human concreteness ratings for 40,000 words are concatenated into an MLP, which reports 0.755 F1 on VUA verbs and 0.718 on All-POS. That design treats concreteness as a fixed input feature. We treat it as a latent quantity that the network must estimate per token, and the ratings act only as a constraint on its averages.
Research Program
The backbone is a RoBERTa encoder in the reading-comprehension setup of DeepMet. DeepMet pairs the sentence with a query word and adds fine-grained part-of-speech tags and a recall-boosting preference parameter. We keep the query formulation and drop the tags. On top of the encoder we add two small heads.
The first head, $g(v)$, maps the static (context-free) embedding of word type $v$ to a concreteness score. It is fit to the Brysbaert-style ratings $r_v$ so that it generalises to unrated words. The second head, $c(i,s)$, maps the contextual representation of token $i$ in sentence $s$ to a concreteness score. The second head never sees token-level labels. Its only supervision is that, over the set $O_v$ of sampled occurrences of type $v$ in a large unlabeled corpus, its average must match the rating:
$$ \mathcal{L}_{\text{agg}} = \sum_{v} \Big( \tfrac{1}{|O_v|} \sum_{(i,s) \in O_v} c(i,s) - r_v \Big)^2 . $$
This is learning from aggregate labels. Each occurrence is free to depart from the type rating, so long as the occurrences of a type agree on average. The detector's classifier receives the clash feature $d_i = g(v_i) - c(i,s)$ together with the contextual representation. The full objective adds the metaphor cross-entropy to $\lambda \mathcal{L}_{\text{agg}}$, with $\lambda$ tuned on the development set.
One subtlety matters. A word's corpus occurrences include its metaphorical uses, so the average is not purely literal. Human ratings, however, are known to reflect the most salient sense, which is usually the literal one. The constraint should therefore push metaphorical occurrences below the type rating, which is the signal we want. A real risk is that the detection loss turns $c$ into a disguised metaphor score. To guard against this, we will collect human ratings of concreteness in context for about 1,500 VUA tokens, spread across genres and stratified by label. We will report how well $c$ correlates with these ratings, and $\lambda$ must keep that correlation above the correlation of the static rating with the same human ratings.
The second part of the program borrows a validation method from imaging. Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification trained on near-raw scans without skull-stripping or tissue segmentation. It then used occlusion maps (masking parts of the input and measuring the change in output) to show that the network attended to the hippocampus and ventricles, the markers clinicians already use. That paper supplies our verification method. We will occlude context words one at a time and measure the change in $d_i$. If the learned clash is meaningful, the biggest shifts should come from the syntactic arguments that carry the target domain, such as "information" in "digest this information". We will compare these localizations to the source-target annotations produced by Explainable Metaphor Identification Inspired by Conceptual Metaphor Theory, which derives mappings from WordNet hypernym paths. Their method needs an external ontology; ours would recover the target-word evidence from the trained model alone.
The same imaging paper also acts as a constraint. Its shallow plain network (AUC 0.88) matched a deeper residual network (0.87) on a small dataset. Our heads are therefore deliberately small. We do not add the multi-scale convolutional stack that a direct transfer of image models would suggest.
Baselines
The naive reference point is plain RoBERTa token classification on the same splits, with no extra features. This tells us how much any feature engineering contributes on top of a modern encoder. The file contains a tension here. The concreteness paper reports a BERT-large-only baseline of 0.574 F1, whereas transformer systems such as DeepMet reach 80.4. A properly tuned plain baseline may already close much of the gap that features once filled, so we will report it carefully rather than borrow the 0.574 figure.
The strongest existing method is DeepMet, which reports F1 of 80.4% on VUA verbs, 76.9% on VUA All-POS and 74.9% on TOEFL. The 2020 shared task report (A Report on the 2020 VUA and TOEFL Metaphor Detection Shared Task) lists 0.769 as its top VUA system and 0.715 on TOEFL. For TroFi, the strongest reported number in the file is 73.2 F1, from the dependency-graph convolution model of Multi-Task Learning for Metaphor Detection with Graph Convolutional Neural Networks and Word Sense Disambiguation. That model runs graph layers over a pre-parsed syntax tree and jointly learns word sense disambiguation. Its reliance on a frozen external parser makes it the natural test case for the genre-brittleness hypothesis. For MOH-X, no reported result was found on Lacuna for DeepMet or the graph model.
For fairness, we will re-run DeepMet and the graph model rather than quote their numbers. Every system will use the same RoBERTa-large checkpoint where an encoder is involved, the 2020 shared-task splits, the same hyperparameter search budget (an equal number of trials per system), and five seeds. We will report means and standard deviations.
The feature-engineered competitor is the concreteness MLP, re-implemented on the same encoder. It feeds static ratings as inputs at inference.
The key ablation isolates the aggregate constraint. In this variant, the contextual head is regressed directly onto the type rating for every occurrence, so every token is pushed toward the same target. Comparing it with our model tests whether letting tokens depart from the type rating, which is the defining ingredient, actually matters.
We count as beating DeepMet three outcomes. The first is an All-POS F1 gain larger than two seed standard deviations with no inference-time tags or norms. The second is matching DeepMet overall while showing a clearly smaller drop on the conversation and fiction subsets. The third is better zero-shot transfer from VUA to TroFi and MOH-X.
Why not just…?
Why not just feed static concreteness ratings into the detector, as the concreteness MLP does? This is the most serious alternative. Static ratings are cheap and already add measurable F1 in Metaphor Detection using Context and Concreteness. Our bet is that a rating which cannot vary with context misses metaphors whose source word is only moderately concrete, and also misses words absent from the 40,000-word list. The re-implemented MLP on an identical encoder settles this directly. If it matches our model on overall F1, on unrated words and on the hard genres, the contribution of a learned contextual estimate is nil. We would then report the result as a negative finding, together with the human-rating correlation analysis.
Why not just fine-tune RoBERTa and let it learn whatever it needs? It may well be that fine-tuning alone already captures concreteness clash implicitly. The large gap between the 0.574 BERT-only baseline and DeepMet's 80.4 suggests that encoder quality and tuning dominate. We test this in two ways. The plain baseline is run under an equal tuning budget. We also fit a linear probe of the plain model's hidden states against the human contextual ratings. If the plain model already encodes contextual concreteness as well as our head does and matches it on F1, the explicit constraint adds nothing beyond interpretability. We would then present the work as an analysis rather than a method.
Why not just prompt a large language model with the MIPVU annotation guidelines? Using Large Language Models to Perform MIPVU-Inspired Automatic Metaphor Detection reports that guideline prompting struggles with edge cases and consistency compared with supervised fine-tuning. MIPVU is the procedure that asks annotators to compare a word's contextual meaning against its basic meaning, and it is the guideline behind the VUA labels. We will still include a few-shot prompted model on the VUA test set as an external reference. We do not expect it to be competitive at the token level.
Why not just use paragraph context instead of new features? Learning Outside the Box shows that paragraph-level context lifts an ELMo plus gradient-boosting classifier from 60.9 to 66.8 F1. That approach is orthogonal to ours, not a substitute. We will run both our model and DeepMet with sentence and paragraph inputs, so that gains from context length are not mistaken for gains from concreteness.
Proposed Experiments
The main experiment trains all systems on VUA and reports token-level F1 on All-POS and verbs, broken down by the corpus's four genres: academic, news, conversation and fiction. We then evaluate without retraining on TOEFL, TroFi and MOH-X.
The first decisive check is the interaction between genre and pipeline. The degradation from news to conversation should be largest for the graph model, intermediate for DeepMet with part-of-speech tags, and smallest for our model. If the ordering does not appear, the brittleness argument fails.
The second check is the aggregate-constraint ablation. It must show that allowing tokens to depart from the type rating improves F1 and raises correlation with the new human ratings of concreteness in context. If per-occurrence regression does as well, the mechanism is not doing what we claim.
The third check is the occlusion analysis on about 300 VUA metaphors with hand-marked target words. Occlusion should rank the target-domain argument first more often than attention weights or a random context word do. Because the explainable-CMT system reports full source-target mappings as 63.7% accurate by human judgment, we will apply the same human rating protocol to our localizations for a rough comparison.
Finally, a small-data run trains on 10%, 25% and 50% of VUA. This tests whether the dictionary signal used during training acts as a useful prior when labeled metaphors are scarce, which is the depth-versus-sample-size tension the imaging work raises.
References
- Metaphor Detection using Context and Concreteness
- DeepMet: A Reading Comprehension Paradigm for Token-level Metaphor Detection
- Residual and plain convolutional neural networks for 3D brain MRI classification
- Explainable Metaphor Identification Inspired by Conceptual Metaphor Theory
- A Report on the 2020 VUA and TOEFL Metaphor Detection Shared Task
- Multi-Task Learning for Metaphor Detection with Graph Convolutional Neural Networks and Word Sense Disambiguation
- Using Large Language Models to Perform MIPVU-Inspired Automatic Metaphor Detection
- Learning Outside the Box: Discourse-level Features Improve Metaphor Identification