Entity Retrieval for Answering Entity-Centric Questions

Hassan Shavarani, Anoop SarkarNAACL 2025

TL;DR

This paper introduces "Entity Retrieval," a novel method for question answering that bypasses traditional document similarity by directly using salient entities within a question to fetch relevant knowledge base articles. This approach proves more accurate and efficient for entity-centric questions, outperforming standard dense and sparse retrieval methods.

What Problem Does This Paper Address?

Large Language Models (LLMs) are powerful, but their factual reliability can be improved by grounding them with external information, a technique known as Retrieval-Augmented Generation (RAG). Traditional RAG systems rely on matching the question's text to documents in a large index. However, for "entity-centric questions" – those seeking concise facts about specific entities (e.g., "Who is the composer of Swan Lake?") – this similarity-based approach can be suboptimal. The authors highlight that dense retrievers, while popular, can struggle with infrequent entities, and irrelevant retrieved documents can degrade answer quality. They aim to develop a more effective retrieval strategy specifically for these entity-focused queries.

Methods

The core innovation is Entity Retrieval. Instead of searching a vast corpus for documents similar to the question, this method first identifies "salient entities" within the question. It then uses these entities to directly look up corresponding articles in a knowledge base (in this case, Wikipedia). These articles are then truncated to a fixed number of words (W) to serve as augmentation documents for the LLM. This approach shifts the retrieval focus from textual similarity to entity identification.

The paper compares Entity Retrieval against established methods:

  • BM25: A classic sparse retrieval method that ranks documents based on term frequency.
  • DPR (Dense Passage Retrieval): A dense retrieval method using bi-encoders to create vector representations of questions and passages.
  • ANCE (A NCE): Another dense retrieval method that uses hard negatives generated during training for improved discriminative features.

Datasets

The authors evaluate their method on three datasets:

  • EntityQuestions (Sciavolino et al., 2021): A dataset specifically designed for entity-centric questions, derived from common relations and fact triples. For their experiments, they use subsets of the dev and test sets, focusing on entities from the top 500K most linked Wikipedia pages.
  • FactoidQA (Smith et al., 2008): A dataset of question-answer pairs derived from Wikipedia articles.
  • StrategyQA (Geva et al., 2021): A dataset requiring implicit reasoning for binary (Yes/No) answers, which necessitates understanding multiple entities within a question.

Key Insights and Ideas

  • Entities as Pointers: The fundamental idea is that for entity-centric questions, the entities themselves are the most direct pointers to relevant information, rather than the overall textual similarity of the question to a document.
  • Fewer, Better Documents: Entity Retrieval often retrieves fewer documents than traditional methods, but these documents are more targeted and relevant, leading to better performance. This aligns with the finding that irrelevant documents can harm LLM output.
  • Efficiency Gains: By directly looking up entity articles rather than searching massive indices, Entity Retrieval offers significant improvements in speed and resource usage.
  • Entity Linking is Crucial: The performance of Entity Retrieval without explicit entity annotations heavily relies on the accuracy of external entity linking tools.

Key Results

  • Superior Accuracy: Entity Retrieval consistently outperformed BM25, DPR, and ANCE on the entity-centric datasets (FactoidQA and EntityQuestions) in terms of nDCG@k and Retrieval Accuracy. For example, ER1000w (Entity Retrieval with 1000 words) surpassed BM25 by 25% on EntityQuestions.
  • Faster Execution: Entity Retrieval was significantly faster than dense retrieval methods like ANCE. It was also 25% faster than BM25 in response generation, while requiring comparable disk and memory resources to a standard personal computer.
  • Impact of Document Length: While longer retrieved articles (ER1000w) showed strong performance, the paper suggests that beyond a certain length, models can become overwhelmed by noise.
  • Entity Linking Performance: When using the SPEL entity linker, Entity Retrieval achieved competitive results, even surpassing BM25 on EntityQuestions and approaching DPR's performance on FactoidQA, demonstrating its potential even without perfect entity annotations.

Limitations and Future Work

  • Reliance on Entity Linking: The method's effectiveness without explicit annotations is directly tied to the performance of external entity linking tools. The paper acknowledges that their exploration of entity linking methods was not exhaustive.
  • Ambiguous Entities: The impact of ambiguous entities on retrieval quality was not investigated.
  • Knowledge Base Specificity: The research primarily used Wikipedia. Exploring other knowledge bases, especially domain-specific ones like UMLS for medical information, is suggested.
  • Language Bias: The current work is limited to English. Extending to multilingual scenarios is a future direction.
  • Data Biases: The authors inherit biases present in the data and do not explicitly de-bias it, trusting the community to use the code ethically.
  • Reducing Memory Footprint: Future work could focus on reducing the memory requirements of Entity Retrieval.

Summary updated

BibTeX
@inproceedings{shavarani2025entity,
  author = {Hassan Shavarani and Anoop Sarkar},
  title = {{Entity Retrieval for Answering Entity-Centric Questions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure