Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals

Hayato Tsukagoshi, Ryohei Sasano, Koichi TakedaNAACL 2022

TL;DR

This paper investigates how different supervision signals (natural language inference vs. dictionary definitions) influence sentence embeddings derived from BERT. They find that these embeddings capture complementary information, leading to improved performance when combined on semantic similarity and downstream NLP tasks.

What Problem Does This Paper Address?

Sentence embeddings are crucial for many NLP tasks, but it's not always clear what kind of meaning these embeddings capture, especially when derived from different training signals. This paper tackles this by comparing two popular sentence embedding methods, SBERT (trained on Natural Language Inference - NLI) and DefSent (trained on dictionary definitions), to understand how their supervision signals shape their learned representations. They aim to reveal the unique properties of each and explore if combining them can yield superior results.

Methods

The core of the paper involves comparing two fine-tuning strategies for pre-trained language models (like BERT and RoBERTa) to produce sentence embeddings:

  • SBERT (Sentence-BERT): Fine-tuned on Natural Language Inference (NLI) datasets. NLI tasks involve determining the relationship (entailment, contradiction, neutral) between sentence pairs. This training encourages the model to distinguish subtle semantic differences.
  • DefSent (Definition Sentence): Fine-tuned on the task of predicting a word from its definition sentence. This approach aims to capture the compositional meaning of words within a definition.

Both methods use similar architectures and pooling strategies (CLS, Mean, Max) to derive sentence embeddings. The researchers then evaluate these embeddings on Semantic Textual Similarity (STS) tasks and a suite of downstream and probing tasks from the SentEval toolkit. They also experiment with combining the embeddings from SBERT and DefSent using various methods (concatenation, averaging, sequential fine-tuning).

Datasets

The paper utilizes several standard datasets for training and evaluation:

  • NLI Datasets (for SBERT training): Stanford NLI (SNLI) and Multi-Genre NLI (MultiNLI).
  • STS Datasets (for evaluation):
    • STS12-STS16 (a collection of datasets from various sources like news, forums, and Q&A).
  • STS Benchmark (STS-B).
  • SICK-R (SICK-Relatedness). These datasets provide sentence pairs with human-annotated similarity scores.

  • SentEval Tasks (for evaluation):

    • Downstream Tasks: Sentiment classification (MR, CR, SST2), paraphrase prediction (MRPC), question-type classification (TREC), etc.
  • Probing Tasks: Tasks designed to test specific linguistic information captured by embeddings, such as sentence length, word content, and tense.

Key Insights and Ideas

  • Complementary Strengths: SBERT and DefSent embeddings capture different aspects of sentence meaning. SBERT excels at distinguishing subtle semantic differences, particularly in superficially similar sentences, likely due to its NLI training. DefSent, on the other hand, is better at capturing compositional meaning, making it suitable for sentences where word meanings combine to form the overall sense.
  • Supervision Signal Matters: The choice of supervision signal significantly impacts the properties of the resulting sentence embeddings, influencing their performance on tasks that require different types of semantic understanding.
  • Combination is Key: Merging SBERT and DefSent embeddings consistently leads to improved performance on unsupervised STS tasks and SentEval downstream tasks compared to using either method alone. This highlights their complementary nature.
  • Task-Specific Performance: SBERT shows an advantage in tasks where sentiment is crucial (e.g., sentiment analysis), while DefSent performs better on tasks requiring a deeper understanding of word meanings and compositionality.

Key Results

  • STS Performance: Combinations like S+D (SBERT then DefSent fine-tuning) and AVERAGE achieved the best average scores on unsupervised STS tasks, outperforming individual SBERT and DefSent. For instance, with BERT-base, the AVERAGE combination achieved a Spearman's rho of 77.81, while SBERT and DefSent achieved 73.19 and 75.20 respectively.
  • SentEval Downstream Tasks: SBERT performed better on sentiment-related tasks (MR, CR, SST2) and paraphrase prediction (MRPC), suggesting it encodes sentiment and semantic equivalence well. DefSent showed strengths in tasks requiring compositional understanding. Overall, the combined methods generally performed comparably or better than individual models.
  • SentEval Probing Tasks: DefSent embeddings showed more information about words within sentences compared to SBERT, which seemed to focus more on semantic similarity. w/oFT (no fine-tuning) performed best on average for probing tasks, suggesting that fine-tuning might specialize embeddings too much.

Limitations and Future Work

  • Limited Scope: The study focused on two specific fine-tuning methods (SBERT and DefSent) and BERT/RoBERTa models.
  • Future Directions: The authors suggest expanding the analysis to other pre-trained models and sentence embedding methods. They also propose investigating how to effectively combine unsupervised sentence embedding methods with supervised ones, and further exploring the mechanisms behind the performance differences observed in combined models.

Summary updated

BibTeX
@inproceedings{tsukagoshi2022comparison,
  author = {Hayato Tsukagoshi and Ryohei Sasano and Koichi Takeda},
  title = {{Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure