PRIS at TAC 2009: Experiments in KBP Track

Si Li, Sanyuan Gao, Zongyu Zhang, Xinsheng Li, Jingyi Guan, Weiran Xu, Jun Guo

Introduction

The Knowledge Base Population (KBP) track at the 2009 Text Analysis Conference (TAC) presented significant challenges in extracting structured information from vast, unstructured datasets. This paper details the approach taken by the PRIS lab, focusing on two core tasks: Entity Linking (EL) and Slot Filling (SF). Their primary innovation lies in the use of query-specific automatic text summarization to perform cross-document coreference resolution, effectively filtering out noise that typically plagues full-document analysis.

Problem Statement

KBP involves two primary hurdles:

  1. Entity Linking (EL): Resolving ambiguous name mentions (e.g., "John Smith") to a specific entry in a large-scale Knowledge Base (KB) like Wikipedia.
  2. Slot Filling (SF): Automatically extracting attributes (e.g., birth date, headquarters) for specific entities from a diverse corpus including news, blogs, and forum posts.

The scale and linguistic variety of these sources make identifying relevant information difficult, as critical data points are often buried within long, noisy documents.

Methodology

Entity Linking (EL)

The PRIS lab implemented a multi-stage pipeline for Entity Linking. Initially, the Indri Retrieval Toolkit was used to fetch the top candidate documents from the KB. To combat the noise inherent in full-text comparison, the team generated query-specific summaries.

System architecture for Entity Linking and coreference decision.

A simplified Hobbs algorithm was applied for intra-document pronoun resolution to ensure summaries remained contextually rich. The system then compared these summaries using two distinct similarity models: the Vector Space Model (VSM) and the Kullback-Leibler (KL) divergence model. A link was established only if entity types (identified via the Stanford Named Entity Recognizer) matched and similarity scores exceeded predefined thresholds.

Slot Filling (SF)

The Slot Filling system utilized a hybrid approach:

  • Unsupervised Extraction: Relied on "trigger words" (e.g., "born in") to identify potential slot values based on proximity.
  • Semi-supervised Learning: Employed Conditional Random Fields (CRFs) trained on 14 feature types, including POS tags and entity proximity.

The system prioritized unsupervised results, falling back to CRF-based predictions when necessary.

Key Results and Analysis

Entity Linking Performance

The initial submission to the TAC competition achieved a Micro-average of 0.3015. However, post-competition optimization of similarity thresholds led to a significant performance boost.

Performance results for Entity Linking using the KL divergence model after threshold optimization.

As shown in the refined results, the KL divergence model achieved a Micro-average of 0.6734 and a Macro-average of 0.7316. The researchers found that treating documents as probability distributions (KL divergence) provided a more robust signal for identity than simple vector overlap (VSM).

Slot Filling Performance

In the Slot Filling task, the system achieved an SF-value score of 0.462 for the top 5 results. The unsupervised trigger-word method proved more reliable than the CRF models, likely due to a domain mismatch between the KB-derived training data and the web-based test data.

Insights and Future Directions

The core insight of this work is that summarization acts as a powerful noise filter. By focusing on sentences directly relevant to the query and resolving internal pronouns, the system creates a cleaner "fingerprint" for entity matching.

However, limitations remain. Retrieval precision at the candidate generation stage is a bottleneck, and automatic summaries can still be redundant if source sentences are overly long. Future work will focus on more sophisticated NLP techniques to handle the informal linguistic styles found in web documents and improving the alignment between training and test domains for slot filling.

Summary updated

BibTeX
@inproceedings{li2009pris,
  author = {Si Li and Sanyuan Gao and Zongyu Zhang and Xinsheng Li and Jingyi Guan and Weiran Xu and Jun Guo},
  title = {{PRIS at TAC 2009: Experiments in KBP Track}},
  booktitle = {TAC},
  year = {2009},
  url = {https://openreview.net/forum?id=VZEXFUfUML},
}
Figure