Semantic Matching from Different Perspectives

Weijie Liu, Tao Zhu, Weiquan Mao, Zhe Zhao, Weigang Guo, Xuefeng Yang, Qi JuarXiv 2022 · arXiv

Abstract

Traditional semantic similarity benchmarks treat the relationship between two text segments as a single scalar value. This paper argues that similarity is multi-faceted and introduces the Multi-Perspective Text Similarity (MPTS) dataset, which evaluates similarity across 12 distinct movie genre perspectives. We evaluate standard transformer-based bi-encoders and cross-encoders, finding that while cross-encoders achieve high accuracy, bi-encoders—essential for efficient retrieval—struggle significantly to capture these nuanced relationships.

Introduction

In Natural Language Processing (NLP), determining if two sentences are "similar" is a foundational task. However, similarity is often subjective and dependent on the observer's "lens." For example, two movie descriptions might be highly similar in terms of genre (e.g., both are Sci-Fi) but entirely different in emotional tone or literal content. Existing benchmarks like STS or QQP fail to capture this complexity. To address this, we introduce the MPTS dataset and propose architectural modifications to handle multi-perspective matching.

The Multi-Perspective Text Similarity (MPTS) Dataset

The MPTS dataset is derived from movie plot summaries on IMDB. It consists of 12,734 pairs labeled across 12 perspectives corresponding to movie genres (e.g., Action, Comedy, Drama). A pair is considered similar from a specific perspective if both movies share that genre tag. This setup explicitly formalizes the idea that two items can be simultaneously similar and dissimilar depending on the criteria used.

Example of how text similarity varies across different semantic perspectives

Methodology

We adapt two standard transformer paradigms for multi-perspective tasks by adding a multi-perspective linear layer—essentially $N$ separate dense layers that project embeddings into $N$ different perspective-specific outputs:

  1. Bi-encoders (Retrieval Mode): These models encode sentences into vectors independently. Similarity for each perspective is calculated using cosine similarity between the perspective-specific embeddings.
  2. Cross-encoders (Rerank Mode): These models feed a pair of sentences into the encoder together, allowing for deep interaction. The multi-perspective layer outputs $N$ similarity scores directly through a Sigmoid function.

Experimental Results

Our evaluation reveals a stark performance gap between the two architectures. Cross-encoders, which benefit from token-level interaction between the two sentences, achieve F1-scores around 90%. In contrast, bi-encoders peak at approximately 65%, even when using optimized pooling strategies.

Comparison of F1-scores between bi-encoders and cross-encoders across different pooling strategies

We also observed that bi-encoders are highly sensitive to implementation details:

  • Pooling Strategy: While cross-encoders are robust to pooling choices, bi-encoders perform significantly better when using the average of the last few layers (e.g., Last2-Avg) compared to the standard [CLS] token, which yielded the lowest results.
  • Temperature Sensitivity: The temperature hyperparameter in the loss function is critical for bi-encoder convergence, with an optimal value found at 0.5; values too high or low lead to significant performance degradation.

Conclusion

The MPTS benchmark highlights a critical weakness in current retrieval models. While cross-encoders handle multi-perspective similarity well, they are too computationally expensive for large-scale applications. The poor performance of bi-encoders suggests that standard sentence embeddings do not yet capture the multi-faceted nature of semantics required for sophisticated search and recommendation systems. Future research should focus on improving bi-encoder representations to bridge this performance gap.

Summary updated

Abstract

In this paper, we pay attention to the issue which is usually overlooked, i.e., \textit{similarity should be determined from different perspectives}. To explore this issue, we release a Multi-Perspective Text Similarity (MPTS) dataset, in which sentence similarities are labeled from twelve perspectives. Furthermore, we conduct a series of experimental analysis on this task by retrofitting some famous text matching models. Finally, we obtain several conclusions and baseline models, laying the foundation for the following investigation of this issue. The dataset and code are publicly available at Github\footnote{\url{https://github.com/autoliuweijie/MPTS}
BibTeX
@misc{liu2022semantic,
  author = {Weijie Liu and Tao Zhu and Weiquan Mao and Zhe Zhao and Weigang Guo and Xuefeng Yang and Qi Ju},
  title = {{Semantic Matching from Different Perspectives}},
  year = {2022},
  eprint = {2202.06517},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.06517},
}
Figure