The Lexical Memorization Bias in Semantic Relation Learning

83 concepts54 papers1 proposal

Consider the task of teaching a machine to recognize a semantic relationship between two words: identifying that a robin is a bird (hypernymy), or that carelessness leads to accidents (causality). On paper, modern neural networks achieve stellar performance on these benchmarks. However, a deeper look reveals a fundamental flaw: the models are often not learning the relationship between the words at all. Instead, they exploit a statistical loophole known as lexical memorization or prototypicality bias.

Rather than learning an abstract relational function $R(x, y)$, supervised models frequently learn to recognize that a specific word $y$ is a highly frequent, prototypical target in the training set. If a dataset repeatedly includes bird or animal as a positive hypernym, a standard classifier learns a shortcut: "If the target word is bird, predict TRUE." The model effectively ignores the source word, failing to develop a genuine understanding of semantic entailment.

The Anatomy of the Shortcut

Why do models prefer rote memorization over relational logic? From an optimization perspective, memorizing the properties of individual words offers a path of least resistance compared to computing abstract, directional transformations across a sparse embedding space.

In a foundational study, Omer Levy et al. demonstrated that supervised distributional models often predict relationships based solely on the presence of these common target words, regardless of the identity of the source word (Do Supervised Distributional Methods Really Learn Lexical Inference Relations?). This "target word pollution" means that the model's internal representations fail to capture the underlying semantic link.

This shortcutting behavior is not restricted to static word vectors; it persists in deep contextualized architectures. Abhilasha Ravichander et al. probed BERT's understanding of hypernymy and discovered that its performance collapses when faced with simple variations like pluralization or novel word pairs (On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT). The representations remain tied to specific lexical pairs seen during training rather than encoding a universal functional representation of the relation.

Standard Evaluation (Overlapping Vocabulary):
Training: (apple, fruit), (car, vehicle)
Testing:  (pear, fruit) -> Model predicts TRUE (exploiting the frequent target "fruit")

Lexical Split Evaluation (Disjoint Vocabulary):
Training: (apple, fruit), (car, vehicle)
Testing:  (oak, tree)    -> Model fails (cannot rely on memorized target words)

This vulnerability manifests across a wide spectrum of natural language tasks:

Diagnosing the Bias via Lexical Splits

To expose this reliance on rote lookup, researchers utilize lexical splits. In a standard random dataset split, a target word like fruit might appear in both the training and testing sets paired with different items. This allows the model to score highly by relying on its memorized target bias.

A lexical split partitions the dataset so that the training and evaluation vocabularies are strictly disjoint. When models are evaluated on these strict splits, their performance frequently collapses to near random chance. As Vered Shwartz et al. demonstrated, this performance drop reveals that standard architectures fail to generalize to unseen vocabularies because they have tracked individual word frequencies rather than learning generalizable relational principles (CogALex-V Shared Task: LexNET - Integrated Path-based and Distributional Method for the Identification of Semantic Relations).

Mechanisms for Relational Generalization

To force models past lexical memorization and toward true relational reasoning, several architectural and training strategies have been developed.

1. Integrating Path-Based Evidence

Purely distributional models look at words within broad contexts, making them highly susceptible to frequency biases. In contrast, path-based approaches examine the literal lexico-syntactic paths—the dependency bridges—that connect two words within a sentence.

Vered Shwartz et al. showed that incorporating path-based evidence forces the model to evaluate the explicit, structural interaction between two words (Path-based vs. Distributional Information in Recognizing Lexical Semantic Relations). By focusing on how words connect in context rather than how often they appear, this hybrid approach mitigates the popularity bias.

2. Geometric and Angular Constraints

Instead of allowing a classifier to draw arbitrary decision boundaries around high-frequency target vectors, researchers introduce geometric constraints that isolate the relational vector space.

Chengyu Wang et al. introduced SphereRE, which maps term pairs into a hyperspherical embedding space (SphereRE: Distinguishing Lexical Relations with Hyperspherical Relation Embeddings). By optimizing for the angular separation of relation triples rather than the absolute positions of individual words, the model is forced to capture the relational logic between entities.

Similarly, Oren Barkan et al. designed a dual-objective loss function that maps relations into distinct sub-spaces (Within-Between Lexical Relation Classification). This prevents the system from ignoring the source word and ensures that classification depends on the semantic connection between both terms.

3. Structural Regularization and Bilinear Mapping

Standard training objectives often encourage models to memorize specific text strings sequentially, leading to phenomena like the "reversal curse" (the inability to infer "B is A" after learning "A is B").

To address this structural limitation, Dong-Kyum Kim et al. demonstrated that enforcing a bilinear relational structure within the hidden layers allows models to naturally encode symmetrical and directional facts (Bilinear representation mitigates reversal curse and enables consistent model editing). This geometric regularization prevents rote string memorization and enables consistent generalization to unseen but logically necessary facts.

4. Explicit Negative Sampling and Inversion

Another approach alters the data distribution to break the predictive power of prototypical labels. Enrico Santus et al. demonstrated that training models on inverted word pairs (e.g., presenting both positive hypernym pairs and their reverses) effectively neutralizes the prototypicality bias (ROOT13: Spotting Hypernyms, Co-Hyponyms and Randoms). This training paradigm forces the model to identify the specific directional relationship between the two terms rather than relying on the inherent properties or frequencies of individual words.

5. Grounding in External Definitions

When a model encounters a rare or long-tail word, it lacks the distributional history required to perform lookup or exploit memorized patterns. Wenpeng Yin 0001 et al. bypassed this limitation by augmenting word embeddings with explicit dictionary definitions through an attentive neural network (Term Definitions Help Hypernymy Detection). By grounding terms in their literal meanings, the model can infer underlying semantic relationships for completely novel vocabulary, bridging the relational generalization gap.

Key Foundational Work

Accurate Semantic Class Classifier for Coreference Resolution (2009) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 60 papers
  2. Version 1 · 2026-10-04 · 91 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_lexical_memorization_4b6b9459,
  title = {lexical memorization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lexical-memorization/rdr_4b6b9459ca4a45f2bf801e445fa19ce2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_lexical_memorization_4b6b9459_v2,
  title = {lexical memorization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lexical-memorization/rdr_4b6b9459ca4a45f2bf801e445fa19ce2/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake