Comparing Convolutional Neural Networks to Traditional Models for Slot Filling

Heike Adel, Benjamin Roth, Hinrich Schütze

Abstract

We address relation classification in the context of slot filling, the task of finding and evaluating fillers like "Steve Jobs" for the slot X in "X founded Apple". We propose a convolutional neural network which splits the input sentence into three parts according to the relation arguments and compare it to state-of-the-art and traditional approaches of relation classification. Finally, we combine different methods and show that the combination is better than individual approaches. We also analyze the effect of genre differences on performance.

Introduction

Slot Filling (SF) is a critical task in knowledge base population, requiring systems to automatically extract structured information about entities from unstructured text. A central challenge in SF is relation classification: determining if a sentence containing two entities expresses a specific relationship (e.g., "founder of"). Traditional approaches have relied on rigid pattern matching or feature-engineered Support Vector Machines (SVMs). However, these methods often struggle with linguistic variability. This paper isolates the relation classification step to evaluate how modern neural architectures, specifically Convolutional Neural Networks (CNNs), compare to and complement traditional methods in a "noisy" real-world setting.

Methods

The study evaluates three primary methodologies for relation classification:

  1. Pattern Matching (PAT): Utilizing patterns derived from distant supervision and Universal Schema relations.
  2. Support Vector Machines (SVMs): Implementing both Bag-of-Words and skip n-gram features to handle structural variations.
  3. Contextwise Convolutional Neural Networks (CNNs): The authors propose a novel "Contextwise" architecture. Unlike standard Piecewise CNNs that convolve the entire sentence before splitting, this model splits the sentence into three distinct regions—left, middle, and right context—relative to the relation arguments before the convolution layer.
Architecture of the proposed Contextwise CNN model, showing the split into three context regions before convolution.

This pre-convolution splitting forces filters to specialize in regional features without being influenced by the specific argument tokens. The model also employs weight sharing across these regions to maintain a manageable parameter count and improve robustness. Finally, a Model Combination (CMB) approach integrates scores from all three methods via linear interpolation.

Datasets and Evaluation

To rigorously test these models, the authors developed a new benchmark by cleaning approximately 39,000 instances from the 2012–2014 TAC Shared Task evaluations. Training data was generated through distant supervision, matching Freebase tuples against large-scale corpora (Wikipedia, NYT, Clueweb). The evaluation set includes diverse genres, specifically Newswire and Web/Forums, to test domain generalizability.

Key Results

The experimental results demonstrate that the Contextwise CNN significantly outperforms the standard Piecewise baseline, increasing $F_1$ from 0.52 to 0.60 on the development set. While the CNN generalizes well due to word embeddings, the SVM with skip n-grams proved surprisingly competitive, often matching or slightly exceeding the CNN's performance on the evaluation set.

F1 scores for various models across different relation types on development and evaluation sets.

The most robust performance was achieved by the Model Combination (CMB). By merging the strengths of pattern matching, SVMs, and CNNs, the CMB model reached an average $F_1$ of 0.53 on the evaluation data, a substantial improvement over the traditional Mintz++ baseline of 0.41.

Discussion and Conclusion

The study highlights that while neural networks offer superior generalization, traditional methods like SVMs remain highly relevant in relation classification. The "genre gap" remains a significant hurdle; models trained on news data see a performance drop when applied to web forums, suggesting that future research should prioritize domain adaptation. Furthermore, the high correlation (0.89) between this benchmark and end-to-end system performance validates the importance of optimizing the relation classification component for effective slot filling. Future work will explore incorporating linguistic structures, such as dependency parsing, into the CNN architecture.

Summary updated

Abstract

We address relation classification in the context of slot filling, the task of finding and evaluating fillers like "Steve Jobs" for the slot X in "X founded Apple". We propose a convolutional neural network which splits the input sentence into three parts according to the relation arguments and compare it to state-of-the-art and traditional approaches of relation classification. Finally, we combine different methods and show that the combination is better than individual approaches. We also analyze the effect of genre differences on performance.
BibTeX
@inproceedings{adel2016comparing,
  author = {Heike Adel and Benjamin Roth and Hinrich Schütze},
  title = {{Comparing Convolutional Neural Networks to Traditional Models for Slot Filling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=H14obQ-_ZB},
}
Figure