DefSent: Sentence Embeddings using Definition Sentences

Hayato Tsukagoshi, Ryohei Sasano, Koichi TakedaACL 2021

TL;DR

This paper introduces DefSent, a novel method for generating sentence embeddings by leveraging word definitions from dictionaries. Unlike previous methods that rely on large, language-specific Natural Language Inference (NLI) datasets, DefSent offers broader language applicability and achieves comparable or superior performance on semantic textual similarity and SentEval tasks.

What Problem Does This Paper Address?

Creating high-quality sentence embeddings, which represent sentences as dense vectors, is crucial for many NLP tasks like semantic textual similarity (STS). While recent advances have leveraged Natural Language Inference (NLI) datasets (e.g., SNLI, Multi-Genre NLI) to fine-tune models like BERT and RoBERTa into powerful sentence encoders (e.g., Sentence-BERT), a significant limitation exists: these NLI datasets are primarily available in English. This severely restricts the applicability of these state-of-the-art methods to other languages, requiring the costly and time-consuming process of creating new NLI datasets for each target language. The authors aim to overcome this language barrier by proposing a method that utilizes readily available resources – dictionaries.

Methods

DefSent's core idea is to train a sentence embedding model by predicting the word from its definition sentence. This is inspired by the observation that a definition sentence and the word it represents share a semantic connection.

The method builds upon pre-trained language models like BERT and RoBERTa. Specifically, it takes a definition sentence, feeds it into the pre-trained model to obtain contextualized word embeddings, and then pools these embeddings to create a sentence embedding. This sentence embedding is then used as input to a "word prediction layer." This layer is essentially the same mechanism used during BERT's pre-training for masked language modeling, where the model learns to predict masked tokens.

The key innovation is how this word prediction layer is used for fine-tuning. Instead of training a new classifier, DefSent leverages the existing word prediction capability. The model is fine-tuned to maximize the probability of predicting the correct word given its definition sentence embedding. Importantly, the parameters of this word prediction layer are kept fixed during fine-tuning, simplifying the process and ensuring that the learned sentence embeddings are semantically aligned with the contextualized embeddings of the target word.

The authors explore three pooling strategies for generating sentence embeddings: CLS (using the special [CLS] token's embedding), Mean (averaging all word embeddings), and Max (taking the maximum over all word embeddings).

Datasets

The primary dataset used for training and evaluating DefSent is derived from the Oxford Dictionary dataset. This dataset consists of pairs of words and their definition sentences. For evaluation, the authors used standard benchmarks:

  • Semantic Textual Similarity (STS) tasks: STS12-16, STS Benchmark, and SICK-Relatedness. These datasets contain sentence pairs with human-assigned similarity scores.
  • SentEval toolkit: This toolkit aggregates various NLP tasks, including classification and sentence similarity, to evaluate the general quality of sentence embeddings.

The authors note that their method is designed to work with dictionaries, which are available in many languages, thus addressing the limitation of language-specific NLI datasets.

Key Insights and Ideas

  • Leveraging Dictionaries for Universal Embeddings: The core insight is that dictionaries, which are widely available across languages, can serve as a valuable resource for training sentence embedding models, bypassing the need for language-specific NLI datasets.
  • Semantic Alignment via Word Prediction: By training the model to predict the word from its definition, DefSent implicitly forces the sentence embeddings to capture the semantic essence of the word. This is a more direct and potentially more robust approach than relying solely on NLI tasks.
  • Efficiency and Broad Applicability: DefSent's reliance on pre-trained models and fixed word prediction layers makes it computationally efficient and easily adaptable to new languages by simply swapping out the dictionary.
  • Performance Parity with NLI-based Methods: The results demonstrate that DefSent can achieve performance comparable to, and in some cases better than, methods trained on massive NLI datasets, validating its effectiveness.

Key Results

  • Word Prediction: DefSent achieved promising results in predicting words from their definitions. For instance, with BERT-large and the CLS pooling strategy, the model achieved a top-1 accuracy of 23.88% and an MRR of 0.3587. Larger models (BERT-large, RoBERTa-large) consistently showed better performance.
  • Semantic Textual Similarity (STS): DefSent models performed comparably to Sentence-BERT models on STS tasks, even with significantly less training data (DefSent used about 5% of the data size of Sentence-BERT). DefSent-RoBERTa models showed particularly strong performance on the STS Benchmark.
  • SentEval Tasks: DefSent-RoBERTa-large achieved the best average score across SentEval tasks, outperforming Sentence-BERT-based methods. This indicates that DefSent embeddings capture useful information for a wide range of NLP tasks.

Limitations and Future Work

  • Out-of-Vocabulary (OOV) Words: The paper acknowledges that DefSent, by relying on the pre-trained word prediction layer of BERT/RoBERTa, cannot directly handle out-of-vocabulary words in a straightforward manner. While they filtered OOV words for their experiments, future work could explore methods to substitute or handle these.
  • Language Expansion: The authors explicitly state their intention to evaluate DefSent's performance on languages other than English and on a broader range of downstream tasks, such as document classification.
  • Model Architecture and Size: Future research will also investigate how model architecture and size influence the embeddings generated by DefSent.
  • Analysis of Embeddings: Further analysis of the relationship between DefSent embeddings and contextualized word embeddings in the semantic vector space is planned.

Summary updated

BibTeX
@inproceedings{tsukagoshi2021defsent,
  author = {Hayato Tsukagoshi and Ryohei Sasano and Koichi Takeda},
  title = {{DefSent: Sentence Embeddings using Definition Sentences}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure