TL;DR
This paper introduces an Entity-based Translation Language Model (ETLM) to improve Question Answering (Q&A) retrieval by explicitly incorporating semantic entities. By leveraging Wikipedia and Q&A archives, ETLM addresses the lexical gap and context-agnostic issues of traditional translation models, leading to significantly better retrieval performance.
What Problem Does This Paper Address?
Community-based Q&A portals like Yahoo! Answers have become massive repositories of information. The challenge in Q&A retrieval is to accurately match a user's question to relevant question-answer pairs within these archives. Traditional approaches often rely on statistical translation models to bridge the "lexical gap" – the difference between words in a user's query and the words in the Q&A pairs. However, these models have limitations:
- Dependence on quality corpus: They require large, high-quality parallel corpora, which are often scarce.
- Noise and generic translations: They can be sensitive to noise in the data and produce translations that are too general.
- Context-agnostic: They treat words in isolation, failing to capture semantic nuances. For example, the word "Blizzard" can refer to a snowstorm or a game company, and without context, translation models might not pick the correct meaning.
This paper aims to overcome these limitations by introducing entities as a primary component in the retrieval process.
Methods
The core contribution is the Entity-based Translation Language Model (ETLM). ETLM extends traditional translation models by incorporating entity information.
- Entity Annotation: The system first annotates both user queries and Q&A pairs with entities. This is done using an entity annotation system that leverages Wikipedia for disambiguation. The process involves identifying mentions, finding the best disambiguation for each mention, and optionally pruning low-confidence annotations.
- Entity-based Translation Language Model (ETLM): ETLM models the probability of translating a token span from a query to a token span in a document, explicitly considering entities. It uses a source-channel framework similar to IBM Model 1 but incorporates entity annotations. This allows for context-aware expansions by leveraging entity information on both the query and document sides.
- Learning Entity-Word Relationships: The paper proposes novel ways to learn relationships between entities and terms. This involves organizing monolingual parallel corpora and leveraging external resources like Wikipedia. Two main configurations are explored:
- ETLMqa: Learns translation probabilities from the Q&A corpus itself.
- ETLMwiki: Leverages Wikipedia to learn entity-entity and entity-word relationships.
- Combining Models: The paper also proposes combining ETLMqa and ETLMwiki using linear interpolation to benefit from the strengths of both sources.
Datasets
- Q&A Corpus: A large-scale dataset from Yahoo! Answers, comprising approximately 5 million Q&A pairs, was used for training and evaluation.
- Wikipedia: Used as an external knowledge base for entity annotation and learning entity-word relationships.
- Manual Annotations: 8.3K manual annotations were collected from 1315 Q&A pairs for evaluating the entity annotation system.
Key Insights and Ideas
- Entities as a Semantic Bridge: Explicitly incorporating entities into translation models provides a more robust way to bridge the lexical gap than word-based methods alone.
- Context is Crucial: Entity disambiguation allows for context-aware query expansion, preventing topic drift and improving relevance.
- Wikipedia as a Rich Resource: Leveraging Wikipedia's vast knowledge base significantly improves the estimation of entity-word and entity-entity relationships, leading to better retrieval performance.
- Combining Sources is Beneficial: Combining models trained on Q&A data (ETLMqa) and Wikipedia (ETLMwiki) yields superior results, suggesting that different sources capture complementary information.
- Entity Annotation Quality Matters: The performance of ETLM is sensitive to the quality and aggressiveness of entity annotations, highlighting the importance of robust disambiguation.
Key Results
- Significant Improvement over Baselines: ETLM configurations consistently outperformed traditional models like VSM, OKAPI BM25, and even advanced translation models like CTM.
- ETLMwiki Outperforms ETLMqa: The model trained using Wikipedia (ETLMwiki) performed better than the one trained solely on Q&A data (ETLMqa), indicating the value of external knowledge.
- ETLM.combo Achieves Best Performance: Combining ETLMqa and ETLMwiki (ETLM.combo) achieved the highest performance across all evaluated metrics (MAP, MRR, R-Prec, Prec@5, Prec@10). For example, ETLM.combo achieved a MAP of 0.427, a significant improvement over CTM's 0.361.
- Robustness to Noise: ETLM models were shown to be more robust to noise in Q&A data compared to models that rely solely on compacting noisy translations.
- Impact of Annotation: Even with minimal query annotations, ETLM performance was competitive with CTM, and performance generally increased with more aggressive annotation up to a certain point.
Limitations and Future Work
- Sparsity in ETLMqa: The ETLMqa model, particularly ETLMqa', suffered from count sparsity when estimating entity-entity translation probabilities, especially when non-linked tokens were removed.
- Wikipedia's Limitations: While Wikipedia is a rich resource, the paper notes that embedding other measures that better capture relationships from Wikipedia could further improve performance.
- Future Work: The authors suggest exploring more powerful context-dependent interpolation techniques for combining models and investigating other measures for capturing relationships from Wikipedia. They also highlight the potential for using entity annotations to improve Q&A retrieval in other contexts.
Summary updated