A Translation Model for Sentence Retrieval

Vanessa Murdock, W. Bruce CroftEMNLP 2005

TL;DR

This paper introduces a novel translation model for monolingual sentence retrieval, significantly outperforming traditional query likelihood methods. They achieve this by developing innovative techniques to construct parallel corpora and by incorporating document context smoothing, leading to more accurate sentence retrieval for tasks like question answering and novelty detection.

What Problem Does This Paper Address?

The core problem this paper tackles is improving the accuracy of sentence retrieval, which is the task of finding relevant sentences within a large collection in response to a user's query. This is a crucial component in applications like question answering, novelty detection, and summarization. Traditional methods often treat sentences as mini-documents, but sentences are much shorter and have sparser term distributions, making it harder to find relevant matches. Furthermore, existing sentence retrieval systems often lack the benefits of passage retrieval, which considers a broader context. The authors aim to bridge this gap by adapting machine translation techniques, specifically translation models, to the monolingual sentence retrieval task, and by exploring how to leverage contextual information around a sentence.

Methods

The paper proposes a translation model for sentence retrieval. This model is based on the idea that a sentence can be "translated" into a query, and vice-versa.

  • Translation Model Estimation: They use IBM Model 1, a standard statistical machine translation model, to estimate translation probabilities between terms.
  • Parallel Corpus Construction: A key challenge for monolingual sentence retrieval is the lack of parallel corpora (pairs of sentences that are translations of each other). The authors propose four methods to construct such corpora:
  • A lexicon learned from a bilingual Arabic-English corpus.
  • Using TREC topic titles and descriptions aligned with relevant sentences.
  • Creating a parallel corpus from WordNet synonym pairs.
  • Deriving an English thesaurus from English-Arabic/Arabic-English lexicons.
  • Monolingual Adaptation: They adapt the translation model for monolingual data by making an adjustment to account for the expected overlap in vocabulary between the query and the sentence.
  • Document Smoothing: To address the sparsity of terms in sentences and leverage contextual information, they explore smoothing techniques. This involves incorporating the term distribution of the document from which the sentence originates, or even the surrounding sentences, into the retrieval model. They compare different smoothing approaches, including Jelinek-Mercer and Dirichlet smoothing.

Datasets

The primary datasets used in this paper are derived from the TREC Novelty Track.

  • TREC Novelty Track Data (2002-2004): This data consists of topic titles and descriptions, and sentence-level relevance judgments for a subset of documents. The documents are news articles from various sources like the Financial Times, Foreign Broadcast Information Service, Los Angeles Times, Xinhua News Service, and Associated Press.
  • Arabic-English Parallel Corpus: Used to derive a bilingual lexicon for training translation models.
  • WordNet: Used to create a parallel corpus of synonym-term pairs.
  • English-Arabic/Arabic-English Lexicons: Used to derive an English thesaurus.

The authors note that the relevance assessments from the TREC Novelty Track only cover a small subset of documents (25 per topic), which limits the evaluation and may lead to lower reported precision values than actual performance.

Key Insights and Ideas

  • Translation Models are Effective for Monolingual Sentence Retrieval: The core insight is that machine translation models, typically used for cross-lingual tasks, can be effectively adapted and applied to monolingual sentence retrieval, outperforming standard query likelihood models.
  • Quality of Parallel Corpus Matters: The performance of the translation model is highly dependent on the quality and size of the parallel corpus used for training. A high-quality bilingual lexicon derived from a large Arabic-English corpus yielded the best results.
  • Context is Crucial for Sentence Retrieval: Smoothing from the local context of a sentence (surrounding sentences or the entire document) significantly improves retrieval performance, suggesting that sentences are rarely isolated units and benefit from their surrounding information.
  • Opinion Queries are Challenging: Opinion-based queries exhibit less vocabulary overlap with relevant sentences compared to event-based queries, making them more challenging for standard retrieval methods and potentially benefiting more from translation models that can capture semantic relationships.

Key Results

  • The proposed translation model, particularly when trained on a high-quality bilingual lexicon (Arabic-English), significantly improves sentence retrieval performance over the query likelihood baseline.
  • Smoothing from the local context of a sentence (e.g., surrounding sentences or the document) dramatically improves retrieval accuracy. Smoothing from the document containing the sentence yielded the best results.
  • For opinion queries, where vocabulary overlap is lower, translation models showed a notable improvement in precision, especially in the top-ranked sentences.
  • While results for description queries were generally lower than for title queries, document smoothing still provided significant improvements.

Limitations and Future Work

  • Limited Relevance Assessments: The authors acknowledge that the TREC Novelty Track relevance judgments are sparse (only 25 documents per topic), which leads to lower reported precision values. They suggest that a more extensive corpus with sentence-level relevance judgments would be needed for a more comprehensive evaluation.
  • Basic Translation Model: The translation model used was a basic one. The authors plan to explore more sophisticated translation models and alignment mechanisms in the future.
  • Lexical vs. Structural Similarity: The current evaluation focuses on lexical similarity. The authors intend to investigate structural similarity as an important measure for future work.
  • Sentence Retrieval for Document Retrieval: Preliminary results suggest that sentence retrieval can be used to improve document retrieval, and they plan to conduct a more extensive investigation into evaluating document similarity and relevance based on sentence-level similarity.

Summary updated

BibTeX
@inproceedings{murdock2005translation,
  author = {Vanessa Murdock and W. Bruce Croft},
  title = {{A Translation Model for Sentence Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure