Incorporating Semantic Textual Similarity and Lexical Matching for Information Retrieval

Hiroki Iida, Naoaki OkazakiPACLIC 2021 · OpenReview

Abstract

Fine-tuning BERT for Information Retrieval (IR) typically requires extensive domain-specific data. This paper proposes Local-Similarity Scoring (LSS), a zero-shot re-ranking method that bridges lexical matching and Semantic Textual Similarity (STS). By comparing contextualized embeddings of overlapping tokens between queries and documents, LSS effectively filters document noise. Experimental results on MS MARCO and Robust04 demonstrate that LSS consistently outperforms BM25 and unsupervised neural baselines without IR-specific fine-tuning.

Introduction

While deep learning models like BERT have significantly advanced Information Retrieval, their effectiveness often depends on massive amounts of labeled data. Fine-tuning a BERT model to surpass traditional lexical baselines like BM25 typically requires tens of thousands of query-document pairs. In scenarios where target-domain data is scarce, off-the-shelf Semantic Textual Similarity (STS) models like Sentence-BERT (SBERT) are often used. However, STS models are generally optimized for comparing texts of similar lengths. In IR, the disparity between short queries and long documents often leads to the semantic signal being drowned out by irrelevant document content. This paper addresses this challenge by introducing a method to leverage pre-trained STS models for IR without requiring target-domain fine-tuning.

Methodology

The authors propose Local-Similarity Scoring (LSS) for the document re-ranking phase. Instead of a global comparison, LSS focuses on the intersection of tokens appearing in both the query and the document.

Architecture of the Local-Similarity Scoring (LSS) framework.

The process involves two main components:

  1. Local-Similarity (LS): A pre-trained STS model extracts contextualized embeddings for overlapping words. The cosine similarity is then calculated between the query token's embedding and the corresponding document token's embedding. Two variants are explored:
  2. Token-based LS: Uses the exact token's embedding.
  3. Pooling-based LS: Averages embeddings within a small window (size 5) to capture broader local context.
  4. Scoring Functions: LS is integrated into three ranking functions:
  5. MAXSIM: Sums the maximum LS values for all overlapping tokens.
  6. MAXSIMIDF: Weights MAXSIM by Inverse Document Frequency (IDF) to prioritize rare, informative words.
  7. BM25-MAXSIM: Uses LS scores to dynamically weight traditional BM25 scores.

Evaluation and Results

The approach was evaluated on three benchmarks: MS MARCO Passage, MS MARCO Document, and Robust04. Initial candidates were retrieved using BM25.

Performance comparison of LSS variants against traditional and neural baselines.

The results indicate that LSS consistently outperforms BM25 and other unsupervised neural baselines like ColBERT. Specifically, BM25-MAXSIM with pooling-based LS proved most effective for document-level tasks, achieving an NDCG@10 of 0.5792 on MS MARCO Document. On the shorter MS MARCO Passage dataset, MAXSIMIDF performed best (MRR@10 of 0.2318), highlighting the importance of explicit Term Frequency (TF) weighting in longer texts.

Key Insights

  • Local vs. Global Similarity: Restricting semantic comparison to overlapping words effectively filters document noise, allowing local similarity to thrive where global similarity fails.
  • Contextualized Lexical Matching: LSS uses contextualized embeddings to ensure that words appearing in both query and document are used in the same sense, adding a layer of semantic verification to lexical matches.
  • Pooling Benefits: Averaging embeddings in a local window (pooling) generally outperformed strict token matching, likely due to the pooling-based training objectives of models like SBERT.

Conclusion and Future Work

LSS provides a robust, zero-shot alternative for IR re-ranking, achieving performance competitive with some fine-tuned models. Its primary limitation is the reliance on exact lexical overlap; it cannot currently account for synonyms or paraphrases that do not share tokens. Future work may involve integrating LSS with query expansion or generation techniques to bridge this semantic gap while maintaining the precision of local contextualized matching.

Summary updated

BibTeX
@inproceedings{iida2021incorporating,
  author = {Hiroki Iida and Naoaki Okazaki},
  title = {{Incorporating Semantic Textual Similarity and Lexical Matching for Information Retrieval}},
  booktitle = {PACLIC},
  year = {2021},
  url = {https://openreview.net/forum?id=0U94t0qzFAx},
}
Figure