Lexical Reference: a Semantic Matching Subtask

Oren Glickman, Eyal Shnarch, Ido DaganEMNLP 2006 · OpenReview

Abstract

Semantic lexical matching is a prominent subtask within text understanding applications. Yet, it is rarely evaluated in a direct manner. This paper proposes a definition for lexical reference which captures the common goals of lexical matching. Based on this definition we created and analyzed a test dataset that was utilized to directly evaluate, compare and improve lexical matching models. We suggest that such decomposition of the global semantic matching task is critical in order to fully understand and improve individual components.

Introduction

In Natural Language Processing (NLP), high-level applications like Question Answering, Summarization, and Information Extraction rely heavily on "semantic matching"—the ability to identify when different linguistic expressions convey the same meaning. Historically, the lexical (word-level) component of these systems has been treated as a "black box." Researchers typically evaluate a lexical matcher based on its marginal contribution to an end-to-end system's performance, rather than its intrinsic accuracy. This indirect evaluation makes it difficult to diagnose failures: is a system failing due to poor word-level matching or a breakdown in higher-level syntactic reasoning? This paper addresses this gap by proposing a standalone definition and evaluation framework for lexical reference.

Defining Lexical Reference

The authors define lexical reference as: A word $w$ is lexically referenced by a text $t$ if there is an explicit or implied reference from a set of words in $t$ to a possible meaning of $w$.

Crucially, lexical reference is broader than simple paraphrasing or synonymy. A text might not contain a direct synonym for a target word, but the concept can be lexically referenced through implication. For example, a sentence describing a "murder trial" lexically references the concept of "death" even if the word itself is absent.

Datasets and Categorization

To facilitate direct evaluation, a new manually annotated dataset was created, derived from the 1st Recognising Textual Entailment (RTE-1) development set. The dataset consists of 580 sentence-word pairs where human annotators reached a consensus. Examples were categorized into three distinct types of reference to capture different levels of semantic complexity:

  • Word: Direct mapping between single words (e.g., murdering $\rightarrow$ died).
  • Phrase: Multi-word expressions referencing a single concept (e.g., call off $\rightarrow$ cancelled).
  • Context: Meaning implied by the overall sentence context rather than a specific trigger word (e.g., a description of a price change $\rightarrow$ increase).
Table 1: Examples of lexical reference categories in the dataset.

Modeling Approaches

Five distinct modeling approaches were evaluated against this benchmark:

  1. WordNet: A baseline utilizing traditional lexical resources to check for synonyms, hyponyms, or derivations.
  2. Distributional Similarity: A model based on the distributional hypothesis, where words appearing in similar contexts are assumed to have similar meanings.
  3. Alignment Model: A probabilistic measure based on word co-occurrence statistics from the Reuters corpus.
  4. Bayesian Model: A contextual measure that treats the text as a document and uses Naïve Bayes to determine if the target word's meaning is present.
  5. Combined Model: A linear interpolation of the WordNet and Bayesian models, designed to leverage both precise lexical knowledge and broader contextual cues.

Key Results

The evaluation revealed that lexical matching is a significant component of the broader entailment problem; an "ideal" lexical reference component would achieve roughly 74% accuracy on the RTE-1 task.

Figure 1: Performance comparison of models using Average Precision (left) and Precision-Recall curves (right).

As shown in Figure 1, the Combined Model (wn+bayes) outperformed all individual models in average precision. Analysis of the subcategories showed that while WordNet and Similarity models were strongest at identifying "Word" level references, the Bayesian model was significantly more effective at identifying "Phrase" and "Context" level references. WordNet achieved a very high precision of 90% but suffered from a low recall of 25%, indicating it is highly accurate when it finds a match but misses the majority of implied references.

Conclusion

By decomposing semantic matching into the subtask of lexical reference, this research provides a clearer path for improving NLP systems. The results demonstrate that word-to-word models and context-to-word models have complementary strengths. Future work should focus on more robust generative models that can better account for domain-specific contexts and optimized supervised learning methods for model combination.

Summary updated

Abstract

Semantic lexical matching is a prominent subtask within text understanding applications. Yet, it is rarely evaluated in a direct manner. This paper proposes a definition for lexical reference which captures the common goals of lexical matching. Based on this definition we created and analyzed a test dataset that was utilized to directly evaluate, compare and improve lexical matching models. We suggest that such decomposition of the global semantic matching task is critical in order to fully understand and improve individual components.
BibTeX
@inproceedings{glickman2006lexical,
  author = {Oren Glickman and Eyal Shnarch and Ido Dagan},
  title = {{Lexical Reference: a Semantic Matching Subtask}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2006},
  url = {https://openreview.net/forum?id=BkNUsGMOZr},
}

Loading related content...

Figure