Improving Related Entity Finding via Incorporating Homepages and Recognizing Fine-grained Entities

Youzheng Wu, Chiori Hori, Hisashi Kawai, Hideki KashiokaIJCNLP 2011 · OpenReview

Abstract

This paper addresses the TREC Related Entity Finding (REF) task by proposing a system that incorporates dependency tree-based similarity, candidate homepage content, and unsupervised fine-grained named entity recognition. By moving beyond coarse-grained Named Entity Recognition (NER) and bag-of-words models, the authors demonstrate significant improvements in ranking accuracy on the TREC 2010 Entity track dataset. The combined system achieves a substantial boost in precision at rank 1 (P@1), increasing from 12.5% in the baseline to 34.0%.

As search engines transition from returning lists of documents to providing direct answers, "entity-oriented search" has become a critical research focus. The TREC Related Entity Finding (REF) track simulates this evolution. In this task, a system is given a source entity and a descriptive narrative—such as "Airlines that currently use Boeing 747 planes"—and must return a ranked list of related entities, typically represented by their official homepages.

Example of a TREC Related Entity Finding (REF) query structure.

Historically, systems have tackled this by using standard NER to extract candidates from web snippets and ranking them using unigram language models based on co-occurrence. However, these methods suffer from three major flaws: standard NER is too coarse (tagging "Continental Airlines" simply as an "Organization"), unigram models ignore syntactic relationships, and systems often ignore the content of the candidate's actual homepage.

Methodology

The authors propose a three-pronged approach to refine the entity ranking phase of the REF pipeline.

Dependency Tree-based Similarity

To capture structural and syntactic relationships, the authors move beyond the "bag-of-words" approach. They parse both the query and the supporting web snippets into dependency trees using the Minipar parser. Similarity is then calculated based on the overlap of these sub-trees, allowing the system to verify if the syntactic relationship between terms in the snippet matches the intent of the query.

Incorporating Homepage Content

The system treats a candidate entity's homepage as vital evidence rather than just a destination. They implement a back-off model that evaluates the relevance of the homepage's main text alongside snippets retrieved directly from the site. If the homepage content heavily discusses the query topic, the entity's ranking is boosted, providing a more robust verification than third-party mentions alone.

Fine-grained Entity Recognition (FG-NER)

To filter out irrelevant candidates, the system extracts the specific fine-grained type requested in the query (e.g., "airline" instead of "organization"). They train an SVM classifier to verify if a candidate matches this type. To avoid the need for manual labeling, they generate training data in an unsupervised manner by parsing Wikipedia definition sentences.

Syntactic patterns extracted from Wikipedia to train the fine-grained entity classifier.

By identifying patterns like "X is a Y" or "X is known as a Y" in Wikipedia's introductory sentences, the system can automatically extract high-quality training pairs (e.g., <Continental Airlines, airline>) for thousands of categories.

Experimental Results

The system was evaluated using the TREC 2010 Entity Track dataset, which utilizes the 503-million-page ClueWeb09 collection. The results showed that the integrated approach significantly outperformed the baseline language model.

  • End-to-End Performance: The combined system achieved an nDCG@R of 0.1865 (up from 0.1336) and a P@1 of 34.0% (up from 12.5%).
  • Entity-Only Performance: When isolating entity name extraction, P@1 reached 50.0%, a 27.5% improvement over the baseline.
  • Ablation Study: Dependency tree-based similarity alone doubled the baseline P@1 score, while the FG-NER component was particularly effective at boosting top-ranked precision to 40.0% when added to the baseline.

Conclusion and Future Work

While the proposed methods significantly improve ranking precision, the authors identify low recall in the initial entity extraction phase as a primary bottleneck, with only 37% of golden answer entities being extracted. Furthermore, the current system struggles with entities found in structured data like tables and lists. Future research will focus on improving extraction from these structured formats and enhancing performance for less famous entities where web mentions are sparse.

Summary updated

BibTeX
@inproceedings{wu2011improving,
  author = {Youzheng Wu and Chiori Hori and Hisashi Kawai and Hideki Kashioka},
  title = {{Improving Related Entity Finding via Incorporating Homepages and Recognizing Fine-grained Entities}},
  booktitle = {IJCNLP},
  year = {2011},
  url = {https://openreview.net/forum?id=GxxkHG2wauB},
}
Figure