Bi-Encoder Matching for Word Sense Disambiguation

22 concepts21 papers1 proposal

When we encounter an ambiguous word like bank in a sentence, our brains instantly use the surrounding context to determine whether it refers to a financial institution or the edge of a river. For decades, Natural Language Processing (NLP) treated this problem—known as Word Sense Disambiguation (WSD)—as a standard multi-class classification task. Models were trained to map a word in context to a fixed set of discrete sense identifiers from a dictionary like WordNet.

However, this classic approach suffers from a severe structural flaw: the label bottleneck. Human language follows a Zipfian distribution, meaning a few word senses are incredibly common, while a vast "long tail" of meanings appears rarely, if ever, in training data. A traditional classifier cannot reliably learn weights for a class it has only seen once or twice, and it fails completely when encountering a new or zero-shot word sense because its output layer is rigidly fixed.

To break through this bottleneck, a powerful paradigm shift has emerged. Instead of classifying a word into an arbitrary category, modern architectures frame WSD as a semantic matching task. By mapping both the word's context and its dictionary definitions (glosses) into a shared vector space, models can determine meaning through geometric proximity.

From Classification to Binary Matching

The intuition behind semantic matching is simple: a sentence containing an ambiguous word and the correct dictionary definition of that word are descriptions of the exact same semantic concept. Therefore, if we can project them into the same embedding space, they should sit close to one another.

Initially, researchers explored this idea using Cross-Encoder architectures. In GlossBERT, Huang et al. reformulated WSD as a sentence-pair classification task. Instead of feeding just the context sentence into a transformer, they concatenated the context sentence directly with a candidate dictionary gloss, separated by a [SEP] token.

[CLS] The bank approved the loan . [SEP] A financial institution .

This formatting allows the transformer's self-attention mechanism to perform deep, word-by-word cross-attention between the context and the definition. Keywords in the sentence (like loan or approved) can directly attend to keywords in the gloss (like financial or institution). Similarly, Yap et al. adopted a joint encoding strategy to optimize a gloss selection objective, treating candidate definitions as a relevance ranking task. As Chen et al. noted, shifting the problem to a binary relationship-matching task allows models to bypass fixed sense inventories entirely, significantly improving generalization to rare meanings.

Yet, Cross-Encoders introduce a massive computational bottleneck. If a target word has 10 possible meanings in a dictionary, the model must run 10 separate, expensive transformer forward passes just to disambiguate a single word. At scale, this makes real-time inference impossibly slow.

The Bi-Encoder Solution: Decoupling for Efficiency

To achieve both high accuracy and computational efficiency, researchers turned to Bi-Encoder (or dual-encoder) architectures. The core innovation of a bi-encoder is the complete decoupling of the context and the definition.

Instead of processing them together, a bi-encoder uses two independent networks:

  1. A Context Encoder ($f$) that processes the sentence and outputs a vector representing the target word's contextual usage.
  2. A Gloss Encoder ($g$) that processes a dictionary definition and outputs a vector representing its core meaning.
Context Sentence ----> [ Context Encoder ] ----> Vector U \
                                                           --> Dot Product / Cosine Similarity
Candidate Gloss  ----> [  Gloss Encoder  ] ----> Vector V /

In Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders, Blevins et al. demonstrated that this architecture can be jointly trained to project both representations into a unified embedding space. Disambiguation then simplifies to a basic geometric operation: calculating the dot product or cosine similarity between the context vector and each candidate gloss vector. The sense with the highest similarity score is selected.

This separation unlocks a massive engineering advantage: caching. Because dictionary definitions are static, every single gloss in an inventory like WordNet can be pre-computed through the gloss encoder and stored in a vector database ahead of time. At inference time, the model only needs to run a single forward pass through the context encoder. The candidate ranking is then reduced to a lightning-fast matrix multiplication against the cached gloss embeddings.

This dual-stage strategy mirrors efficient retrieval architectures used in other complex NLP tasks. For instance, in text-based virtual environments, Urbanek et al. utilized separate vector spaces for environmental context and candidate dialogue responses to allow fast, real-time retrieval via dot products. Similarly, Hu et al. leveraged dual recurrent networks within a generative adversarial framework to match sentences and definitions competitively.

Flattening the Long Tail

By converting WSD from a classification task to a metric learning task, bi-encoders inherently solve the data sparsity problem. The model no longer needs to memorize what "sense #3 of bank" looks like; instead, it learns a universal relationship: how to match contextual usage to text definitions.

This capability allows the architecture to excel in low-shot and zero-shot scenarios. In FEWS, Blevins et al. demonstrated that a bi-encoder trained on a diverse dictionary dataset can accurately disambiguate words it has never seen during training, simply by evaluating the semantic similarity of new definitions.

To further optimize this shared space, several enhancements have been developed:

  • Inventory Alignment: To combat the lack of training examples for rare senses, Yao et al. aligned definitions across multiple dictionaries. By treating different definitions of the same sense as semantically equivalent "surrogates," the authors expanded the training signal without manual annotation.
  • Zipfian Reweighting: Recognizing that standard loss functions favor frequent words, Zhang et al. introduced Z-Reweighting, a training strategy based on Zipf's Law that explicitly adjusts loss contributions to prevent dominant senses from crowding out the vector space of rare meanings.
  • Cross-Lingual Transfer: The bi-encoder framework easily extends across languages. In mBERT-UNI, Zhang et al. mapped low-resource target languages and English definitions into a unified sense representation space, allowing models to perform zero-shot multilingual WSD by matching non-English contexts directly to English glosses.
  • Enriched Representations: To make the gloss embeddings more distinct, Song et al. augmented dictionary definitions with synonyms, hypernyms, and example sentences, providing a richer target for the context encoder to align with.

Beyond Word Senses: Metaphor and Definition Modeling

The core mechanism of bi-encoder matching—comparing a dynamic contextual representation against a grounded semantic reference—has proven highly effective for broader linguistic tasks that require deep contextual awareness.

Metaphor Detection

Detecting a metaphor requires recognizing when a word has drifted away from its literal meaning. Dual-encoder architectures are perfectly suited for this comparison. In EmbodiedBERT, Peng et al. used a dual-encoder framework to simultaneously capture a word's contextual representation and its basic, literal sensorimotor meaning, measuring the semantic distance between them to flag metaphorical usage. Similarly, Li et al. developed FrameBERT, which uses one encoder for standard contextual embeddings and a second encoder fine-tuned on a linguistic database to map words to explicit semantic frames, exposing the gap between literal and contextual application.

Generative Definition and Language Modeling

The inverse of WSD is definition modeling: generating a dictionary definition for a given word in context. In VCDM, Reid et al. utilized a variational bi-encoding approach to explicitly separate a phrase-context pair, ensuring that the generation network isolates the specific word being defined from its surrounding context. For language modeling itself, Ansell et al. introduced PolyLM, which employs a dual-encoder structure and a match loss to ensure that a word's internal sense representation aligns cleanly with the representation predicted solely from the surrounding context, preventing polysemous meanings from conflating into a single vector.

Even in abstractive text summarization, the underlying principle holds. Sheng et al. designed a pseudo-siamese discriminator architecture that uses distinct encoders for source and target texts. By mapping both into a unified semantic space, the model evaluates summary quality based on shared core meaning rather than surface-level token overlap.

Conclusion

The transition from discrete classification to bi-encoder similarity matching fundamentally changed how neural networks handle polysemy. By decoupling context from definition, bi-encoders allow models to learn the underlying language of semantic equivalence. This architecture provides a rare trifecta in machine learning: it eliminates the label bottleneck, generalizes seamlessly to zero-shot instances in the long tail, and maintains high computational efficiency through vector caching.

Key Foundational Work

WSD-GAN: Word Sense Disambiguation Using Generative Adversarial Networks. (2019) — contributed 1 concept elements to this direction.
Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders (2020) — contributed 1 concept elements to this direction.
VCDM: Leveraging Variational Bi-encoding and Deep Contextualized Word Representations for Improved Definition Modeling (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dual_encoder_similarity_scoring_ec0a6c76,
  title = {dual-encoder similarity scoring},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dual-encoder-similarity-scoring/rdr_ec0a6c760a36436baec3ce0443829fb1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dual_encoder_similarity_scoring_ec0a6c76_v1,
  title = {dual-encoder similarity scoring},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dual-encoder-similarity-scoring/rdr_ec0a6c760a36436baec3ce0443829fb1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake