SoochowNLP Team System Description for 2016 KBP Slot Filling and Nugget Detection Tasks

Yu Hong, Yingying Qiu, Zengzhuang Xu, Wenxuan Zhou, Jian Tang, Xiaobin Wang, Liang Yao, Jianmin YaoTAC 2016

Introduction

Information Extraction (IE) tasks like Knowledge Base Population (KBP) present significant challenges in identifying attributes for entities (Slot Filling) and detecting event mentions (Nugget Detection). The SoochowNLP team addressed these challenges in the 2016 KBP competition by introducing a social-network-based verification layer for Slot Filling and an "allied propagation" method for Nugget Detection. Their approach focuses on reducing noise and improving recall by leveraging the target entity's broader social context and semantic neighborhood.

Methodology

Slot Filling via Social Network Verification

The Slot Filling (SF) pipeline aims to find specific attributes, such as "employer" or "spouse," for a given entity. To mitigate noise where names appear near a target entity without a meaningful relationship, the team implemented a multi-stage verification process.

  1. Baseline Extraction: A pattern-based engine identifies candidates using entity types and trigger words.
  2. Trigger Lexicon Expansion: The team utilized FrameNet to expand their trigger word list. By mapping specific words to abstract semantic frames, the system recognizes events described with diverse vocabulary. For instance, the "Giving_Birth" frame includes terms like "produce," "bear," and "sire."
FrameNet expansion for the Giving_Birth frame provides semantically related trigger words.
  1. Social Network Construction: Two networks are built: a Reference Social Network (RSO) from the document containing the query, and an Additional Social Network (ASO) from pseudo-relevant documents.
  2. Verification: Candidate fillers are clustered into Candidate Social Networks (CSOs). A filler is accepted only if its CSO aligns with the RSO or ASO, serving as a contextual "sanity check."

Nugget Detection via Allied Propagation

To improve the detection of event mentions (nuggets), the team introduced Allied Propagation. This method identifies "reliable nuggets" using a Maximum Entropy classifier and then looks for "allied" terms—semantically related words found via FrameNet or Mutual Information. If an allied term appears in a similar context or shares a topic (determined via Latent Dirichlet Allocation), the event label is propagated to it, capturing mentions the initial classifier might miss.

Evaluation and Results

The team evaluated their methods using the LDC2016E63 corpus and historical KBP datasets.

Slot Filling Performance

On the 2013 evaluation pool, the social network verification significantly improved precision and F1 scores. While the baseline precision was 0.21, the "Our best" system reached 0.24, with an F1 increase from 0.34 to 0.38.

Performance comparison of the Slot Filling verification method on the 2013 evaluation pool.

Nugget Detection Performance

The propagation methods (Systems 2 and 3) showed consistent improvements over the baseline. System 3, which incorporated topic-level sentence similarity, achieved the highest F1 scores across most metrics, including a Plain F1 of 50.83 compared to the baseline's 50.33.

Discussion and Future Work

Despite improvements, the team noted that baseline fragility—specifically errors in entity recognition and document retrieval—limited overall performance. Future work will focus on frame disambiguation to handle polysemous trigger words and the use of sentence-level embeddings to capture implicit event mentions (e.g., "owning a happy family" implying marriage).

Conclusion

The SoochowNLP team's 2016 KBP system demonstrates that social context and semantic frames are powerful tools for refining information extraction. By treating an entity's social network as a filter and using allied propagation to expand event detection, they successfully reduced false positives and improved the robustness of their extraction pipeline.

Summary updated

BibTeX
@inproceedings{hong2016soochownlp,
  author = {Yu Hong and Yingying Qiu and Zengzhuang Xu and Wenxuan Zhou and Jian Tang and Xiaobin Wang and Liang Yao and Jianmin Yao},
  title = {{SoochowNLP Team System Description for 2016 KBP Slot Filling and Nugget Detection Tasks}},
  booktitle = {TAC},
  year = {2016},
}
Figure