Improving Slot Filling Performance with Attentive Neural Networks on Dependency Structures

Lifu Huang, Avirup Sil, Heng Ji, Radu Florian

Abstract

Slot Filling (SF) aims to extract the values of certain types of attributes (or slots, such as person:cities_of_residence) for a given entity from a large collection of source documents. In this paper, we propose an effective DNN architecture for SF with the following new strategies: (1) Take a regularized dependency graph instead of a raw sentence as input to the DNN to compress the wide contexts between query and candidate filler; (2) Incorporate two attention mechanisms: local attention learned from the query and candidate filler, and global attention learned from external knowledge bases, to guide the model to better select indicative contexts to determine slot type. Experiments show that this framework outperforms state-of-the-art on both relation extraction (16% absolute F-score gain) and slot filling validation for each individual system (up to 8.5% absolute F-score gain).

1. Introduction: The Long-Distance Challenge

In Information Extraction, Slot Filling (SF) involves identifying specific attributes (fillers) for a given entity (query) across vast document collections. A significant hurdle in SF is the "long-distance" problem. While standard Relation Extraction (RE) datasets like ACE or SemEval often feature entities in close proximity, SF pairs are frequently separated by extensive irrelevant context.

Distribution of word distance between entities across different datasets.

As shown in Figure 1, while approximately 70% of relations in standard datasets have entities separated by at most one word, roughly 46% of Slot Filling pairs are separated by 7 words or more. Standard Deep Neural Networks (DNNs) struggle to capture dependencies across such wide contexts, necessitating a more robust approach to context compression and focus.

2. Methodology: Graph-based CNN with Dual Attention

To address these challenges, we propose a Graph-based CNN architecture that utilizes dependency structures and dual attention mechanisms to focus on indicative contexts.

The proposed Attention-based CNN architecture for Slot Filling.

2.1 Regularized Dependency Graphs

Rather than processing raw word sequences, the model operates on a dependency parse tree. It extracts a regularized dependency graph consisting of the shortest dependency path between the query and the filler, along with their immediate (1-hop) neighbors. This approach effectively "shrinks" the distance between entities, bringing them closer together regardless of intervening clauses.

2.2 Dual Attention Mechanisms

The architecture incorporates two distinct attention layers to guide the CNN:

  • Local Attention: This mechanism analyzes the specific query and candidate filler to determine which parts of the surrounding context are most relevant to their specific relationship.
  • Global Attention (Knowledge Base Attention): This leverages external knowledge from DBPedia and TransE embeddings to create prototypical vectors for each slot type. These prototypes help the model identify "trigger words" (e.g., "owns", "acquired") that signal specific relations, even in noisy environments.

3. Experimental Results

The model was evaluated on the TAC-KBP (2012-2014) datasets, with word embeddings pre-trained on a Wikipedia dump. Key findings include:

  • Relation Extraction Performance: The proposed model achieved a 57.39% Micro F1 score, representing a 16% absolute improvement over previous state-of-the-art "Context-CNN" models (41.31%).
  • Slot Filling Validation: When used to re-classify outputs from existing SF systems, the model provided absolute F-score gains of up to 8.5%.
  • Precision Improvements: The model demonstrated high precision (70.46%), significantly reducing the rate of false-positive relation extractions compared to baseline methods.

4. Discussion and Future Work

While the model significantly advances SF performance, analysis of remaining errors reveals that over 58% are "spurious," often due to conflicting contextual clues. Future work will explore fine-grained entity typing to better distinguish between similar relations (e.g., "city of death" vs. "city of residence") and the integration of external gazetteers like Geonames to provide additional context for entities when document-level information is sparse.

Summary updated

Abstract

Slot Filling (SF) aims to extract the values of certain types of attributes (or slots, such as person:cities\_of\_residence) for a given entity from a large collection of source documents. In this paper we propose an effective DNN architecture for SF with the following new strategies: (1). Take a regularized dependency graph instead of a raw sentence as input to DNN, to compress the wide contexts between query and candidate filler; (2). Incorporate two attention mechanisms: local attention learned from query and candidate filler, and global attention learned from external knowledge bases, to guide the model to better select indicative contexts to determine slot type. Experiments show that this framework outperforms state-of-the-art on both relation extraction (16\% absolute F-score gain) and slot filling validation for each individual system (up to 8.5\% absolute F-score gain).
BibTeX
@inproceedings{huang2017improving,
  author = {Lifu Huang and Avirup Sil and Heng Ji and Radu Florian},
  title = {{Improving Slot Filling Performance with Attentive Neural Networks on Dependency Structures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=OkI029tfw1},
}
Figure