Abstract
This paper addresses the TREC Related Entity Finding (REF) task by proposing a system that incorporates dependency tree-based similarity, candidate homepage content, and unsupervised fine-grained named entity recognition. By moving beyond coarse-grained Named Entity Recognition (NER) and bag-of-words models, the authors demonstrate significant improvements in ranking accuracy on the TREC 2010 Entity track dataset. The combined system achieves a substantial boost in precision at rank 1 (P@1), increasing from 12.5% in the baseline to 34.0%.
Introduction: The Related Entity Finding Task
As search engines transition from returning lists of documents to providing direct answers, "entity-oriented search" has become a critical research focus. The TREC Related Entity Finding (REF) track simulates this evolution. In this task, a system is given a source entity and a descriptive narrative—such as "Airlines that currently use Boeing 747 planes"—and must return a ranked list of related entities, typically represented by their official homepages.
Historically, systems have tackled this by using standard NER to extract candidates from web snippets and ranking them using unigram language models based on co-occurrence. However, these methods suffer from three major flaws: standard NER is too coarse (tagging "Continental Airlines" simply as an "Organization"), unigram models ignore syntactic relationships, and systems often ignore the content of the candidate's actual homepage.
Methodology
The authors propose a three-pronged approach to refine the entity ranking phase of the REF pipeline.
Dependency Tree-based Similarity
To capture structural and syntactic relationships, the authors move beyond the "bag-of-words" approach. They parse both the query and the supporting web snippets into dependency trees using the Minipar parser. Similarity is then calculated based on the overlap of these sub-trees, allowing the system to verify if the syntactic relationship between terms in the snippet matches the intent of the query.
Incorporating Homepage Content
The system treats a candidate entity's homepage as vital evidence rather than just a destination. They implement a back-off model that evaluates the relevance of the homepage's main text alongside snippets retrieved directly from the site. If the homepage content heavily discusses the query topic, the entity's ranking is boosted, providing a more robust verification than third-party mentions alone.
Fine-grained Entity Recognition (FG-NER)
To filter out irrelevant candidates, the system extracts the specific fine-grained type requested in the query (e.g., "airline" instead of "organization"). They train an SVM classifier to verify if a candidate matches this type. To avoid the need for manual labeling, they generate training data in an unsupervised manner by parsing Wikipedia definition sentences.
By identifying patterns like "X is a Y" or "X is known as a Y" in Wikipedia's introductory sentences, the system can automatically extract high-quality training pairs (e.g., <Continental Airlines, airline>) for thousands of categories.
Experimental Results
The system was evaluated using the TREC 2010 Entity Track dataset, which utilizes the 503-million-page ClueWeb09 collection. The results showed that the integrated approach significantly outperformed the baseline language model.
- End-to-End Performance: The combined system achieved an nDCG@R of 0.1865 (up from 0.1336) and a P@1 of 34.0% (up from 12.5%).
- Entity-Only Performance: When isolating entity name extraction, P@1 reached 50.0%, a 27.5% improvement over the baseline.
- Ablation Study: Dependency tree-based similarity alone doubled the baseline P@1 score, while the FG-NER component was particularly effective at boosting top-ranked precision to 40.0% when added to the baseline.
Conclusion and Future Work
While the proposed methods significantly improve ranking precision, the authors identify low recall in the initial entity extraction phase as a primary bottleneck, with only 37% of golden answer entities being extracted. Furthermore, the current system struggles with entities found in structured data like tables and lists. Future research will focus on improving extraction from these structured formats and enhancing performance for less famous entities where web mentions are sparse.
Summary updated