Exploring Classic and Neural Lexical Translation Models for Information Retrieval: Interpretability, Effectiveness, and Efficiency BenefitsL

Leonid Boytsov, J Zico KolterOpenReview 2021 · OpenReview

TL;DR

This paper revisits the classic IBM Model 1 for information retrieval, adapting it with neural networks to work with modern contextual embeddings like BERT. The resulting "neural Model 1" offers a compelling trade-off: it maintains or improves retrieval accuracy and efficiency, is highly interpretable, and crucially, can be made efficient enough to run on a CPU, overcoming BERT's limitations with long documents.

What Problem Does This Paper Address?

Traditional information retrieval (IR) systems often struggle with the "vocabulary gap" – the mismatch between query terms and document terms. The IBM Model 1, a lexical translation model, was proposed to bridge this gap by estimating the probability of query terms being translations of document terms. While effective, Model 1 has limitations: it requires large query sets for training, is less effective than modern deep learning models, and its non-parametric nature can be computationally expensive. Modern deep learning models like BERT excel at understanding context but have a fixed input length, making them inefficient for long documents. This paper aims to combine the strengths of both approaches: the interpretability and term-matching capabilities of Model 1 with the contextual understanding of BERT, while addressing efficiency concerns.

Methods

The core innovation is the integration of a "neural Model 1" layer. This layer acts as an aggregator, taking embeddings from either context-free (e.g., GloVe) or contextualized (e.g., BERT) models and producing translation probabilities.

  • Neural Model 1 Layer: This layer is designed to be interpretable. It computes pairwise similarities between query and document "word pieces" (BERT's subword units) and combines them using a straightforward product-of-sums formula, without learned weights in this specific formulation. This allows for easy pinpointing of which query and document terms contribute most to the score.
  • Context-Free Neural Model 1: This variant uses context-free embeddings and is trained end-to-end. While less effective than BERT-based models, it can be made highly efficient.
  • Contextualized Neural Model 1 (BERT-based): This approach applies the neural Model 1 layer on top of BERT's contextualized embeddings. This allows it to leverage BERT's understanding of word meaning in context. A key advantage here is that this architecture can handle longer documents by processing them in chunks and aggregating the results, overcoming BERT's typical sequence length limitations.
  • Sparsification: A crucial step for efficiency is the "sparsification" of the neural Model 1. This involves precomputing translation probabilities, discarding very small values, and storing the result as a sparse matrix. This allows the model to run efficiently on a CPU.

Datasets

The experiments were conducted on the MS MARCO collections, specifically for passage and document retrieval tasks. These collections are highlighted for their large number of real user queries. The paper also mentions using evaluation sets from TREC 2019/2020 deep learning tracks, which are smaller but thoroughly judged. The MS MARCO data is described as having sparsely judged relevance labels (binary: relevant or not relevant).

Key Insights and Ideas

  • Bridging the Gap with Interpretability: The paper demonstrates that a classic IR concept like Model 1 can be effectively modernized and integrated with deep learning, offering a path to more interpretable neural ranking systems.
  • CPU Efficiency is Achievable: The sparsification technique for the neural Model 1 is a major win. It allows a neural model to run orders of magnitude faster on a CPU than GPU-based BERT rankers, making it practical for first-stage retrieval or scenarios where GPU resources are limited.
  • Overcoming BERT's Length Limitation: By using the neural Model 1 as an aggregator on top of BERT's chunked outputs, the system can effectively process long documents without significant accuracy loss, a common bottleneck for standard BERT models.
  • Fusion Power: Combining the neural Model 1 (especially the context-free version) with traditional BM25 scores yields substantial improvements, outperforming both individual components and other fusion strategies.

Key Results

  • Accuracy: The BERT-based neural Model 1 achieved competitive results, with the "full" document version showing slight improvements over shorter versions, suggesting its effectiveness in handling longer contexts. Fusion of the context-free neural Model 1 with BM25 scores showed significant gains (around 10-17% on MS MARCO test sets) compared to BM25 alone.
  • Efficiency: The sparsified neural Model 1 (NN-Modell-exp) achieved remarkable speedups, running over 10³ times faster than BERT-based rankers on a CPU. This is a critical finding for practical deployment.
  • Interpretability: The product-of-sums formulation of the neural Model 1 layer allows for clear identification of contributing query and document terms, enhancing model interpretability.
  • Leaderboard Performance: The authors report achieving top positions on the MS MARCO document ranking leaderboard in late 2020 using their best neural and non-neural runs, validating their approach.

Limitations and Future Work

  • Context-Free Model Effectiveness: The paper notes that context-free Model 1 rankers, on their own, are not as strong as BERT-based models, necessitating fusion with other methods like BM25.
  • Inconclusive Evidence for Long Documents: While the BERT-based Model 1 shows promise for long documents, the paper mentions that evidence was "somewhat inconclusive" when compared to other aggregator models like CEDR, suggesting that further investigation on different datasets might be needed.
  • Future Work: The authors suggest that further improvements could be made by designing better parameterizations for conditional translation probabilities and exploring other embedding networks. They also hint at the potential for the efficient context-free neural Model 1 to be used in the candidate generation stage of retrieval systems.

Summary updated

Abstract

We study the utility of the lexical translation model (IBM Model 1) for English text retrieval, in particular, its neural variants that are trained end-to-end. We use the neural Model1 as an aggregator layer applied to context-free or contextualized query/document embeddings. This new approach to design a neural ranking system has benefits for effectiveness, efficiency, and interpretability. Specifically, we show that adding an interpretable neural Model 1 layer on top of BERT-based contextualized embeddings (1) does not decrease accuracy and/or efficiency; and (2) may overcome the limitation on the maximum sequence length of existing BERT models. The context-free neural Model 1 is less effective than a BERT-based ranking model, but it can run efficiently on a CPU (without expensive index-time precomputation or query-time operations on large tensors). Using Model 1 we produced best neural and non-neural runs on the MS MARCO document ranking leaderboard in late 2020.
BibTeX
@misc{boytsov2021exploring,
  author = {Leonid Boytsov and J Zico Kolter},
  title = {{Exploring Classic and Neural Lexical Translation Models for Information Retrieval: Interpretability, Effectiveness, and Efficiency BenefitsL}},
  year = {2021},
  url = {https://openreview.net/forum?id=TWxgWQbcic2},
  howpublished = {OpenReview},
}
Figure