Exemplification Modeling: Can You Give Me an Example, Please?

Edoardo Barba, Luigi Procopio, Caterina Lacerra, Tommaso Pasini, Roberto NavigliIJCAI 2021 · DOI

TL;DR

This paper introduces "Exemplification Modeling" (EXMOD), a novel task focused on generating example sentences for word senses given their definitions. They propose a sequence-to-sequence model (EXMAKER) that can produce high-quality, human-comparable examples, significantly boosting performance when used to train Word Sense Disambiguation (WSD) models and even achieving a new state-of-the-art.

What Problem Does This Paper Address?

For decades, the field of Natural Language Processing has grappled with understanding and representing word meanings. A key task is Word Sense Disambiguation (WSD), which aims to identify the correct meaning of a word in a given context. Traditionally, WSD relies on manually curated datasets and predefined sense inventories, which are often incomplete and labor-intensive to create. While "Definition Modeling" has emerged to automatically generate definitions for words, the inverse problem – generating usage examples from definitions – remained largely unexplored. This gap is significant because high-quality examples are crucial for training robust WSD systems and for enriching lexical resources. The authors aim to bridge this gap by introducing the EXMOD task and a method to generate these examples automatically.

Methods

The core of this work is the Exemplification Modeling (EXMOD) task, which takes a set of (lemma, definition) pairs as input and outputs a sentence that uses the lemma with the specified meaning.

The authors propose a two-stage approach:

  1. Sequence-to-Sequence Generation: They employ a BART-based encoder-decoder model, named EXMAKER, to generate a sentence. The input to the model is a concatenation of the lemmas and their definitions, with special tokens marking the boundaries. The target output is the example sentence, with special markers around the words corresponding to the input definitions. This stage is trained using a cross-entropy loss.
  2. Post-processing: After the initial generation, a four-step process refines the output. This includes lemmatization, identifying spans corresponding to input pairs, computing indices of where the lemma appears, and finally, removing the special markers.

For training, they developed a sampling strategy to select appropriate (input, output) pairs from existing corpora. This strategy prioritizes pairs where multiple senses co-occur nearby and are statistically significant, encouraging the model to learn interactions between senses.

Datasets

The paper utilizes several existing lexical resources for training and evaluation:

  • WordNet: A foundational lexical database providing synsets, lemmas, and glosses (definitions).
  • SyntagNet: A manually curated resource of semantic collocations.
  • SemCor: A large sense-annotated corpus for WSD.
  • Princeton WordNet Gloss Corpus (WNG): Contains examples and definitions from WordNet.
  • Oxford Dictionary Dataset (ODE): A large dataset of sentences tagged with senses from the Oxford Dictionary of English.

They also introduce a framework for in vivo evaluation by using the generated examples to train WSD models.

Key Insights and Ideas

  • The Inverse Problem is Solvable: The paper demonstrates that generating usage examples from definitions is a feasible and valuable task, complementing existing definition generation approaches.
  • EXMOD Data Boosts WSD: The generated examples, when used as additional training data for WSD models, significantly improve performance, even surpassing results obtained with manually annotated data alone. This highlights the quality and utility of the generated data.
  • Generalization Across Inventories: EXMAKER can generate examples for senses from an inventory it wasn't trained on, showcasing its generalization capabilities.
  • Handling Multiple Senses: The model can generate examples for up to three senses simultaneously, opening avenues for modeling complex sense interactions.
  • "Silver" Data Quality: The generated "silver" data is remarkably close in quality (fluency and semantic coherence) to human-annotated "gold" data, as confirmed by human evaluation.

Key Results

  • WSD Performance Improvement: Using EXMAKER-generated data, the BERT-based WSD model achieved a 1.3-point boost on Task 1 compared to using only gold data.
  • State-of-the-Art WSD: The state-of-the-art ESCHER model, when trained with EXMAKER data, achieved an unprecedented F1 score of 82.3, surpassing the previous state-of-the-art by 1.6 points.
  • Human Evaluation: Annotators rated EXMAKER's generated sentences with average Likert scores of 4.63 for fluency and 4.82 for semantic coherence, nearly identical to human-curated corpora. Inter-annotator agreement was substantial.
  • Handling Rare/Unseen Senses: The generated data proved particularly beneficial for improving WSD performance on less frequent and unseen senses.

Limitations and Future Work

  • Model Selection: Choosing the best model weights for EXMAKER was challenging due to the open-ended nature of text generation. They addressed this with an unreferenced metric based on latent representations of WordNet senses.
  • Language Expansion: The current work focuses on English. Future work aims to extend the generation of examples to other languages.
  • Input Sense Count: While the model can handle up to three senses, future work plans to enlarge the number of input senses the architecture can handle.

Summary updated

BibTeX
@inproceedings{barba2021exemplification,
  author = {Edoardo Barba and Luigi Procopio and Caterina Lacerra and Tommaso Pasini and Roberto Navigli},
  title = {{Exemplification Modeling: Can You Give Me an Example, Please?}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/520},
}
Figure