TL;DR
This paper introduces a new benchmark for relation classification in the context of "slot filling" and proposes a Contextwise Convolutional Neural Network (CNN) that outperforms previous state-of-the-art neural models. By splitting sentence contexts before processing them, the authors show that neural networks, especially when combined with traditional SVMs and pattern matchers, provide a more robust solution for extracting structured knowledge from messy, real-world text.
What Problem Does This Paper Address?
In the world of Natural Language Processing (NLP), Slot Filling (SF) is the task of automatically populating a database with information about an entity. For example, if the entity is "Apple" and the slot is "founder," the system should find "Steve Jobs" in a text like "Steve Jobs founded Apple."
The "key challenge" here is relation classification: given a sentence containing two potential arguments, does the sentence actually express the relationship we’re looking for? Historically, this has been tackled using rigid patterns or feature-heavy Support Vector Machines (SVMs). However, these models often struggle with the sheer variety of ways humans express the same idea. Furthermore, most slot-filling research evaluates "end-to-end" systems, making it hard to tell if a failure happened because of poor search, bad entity recognition, or a weak relation classifier. The authors aim to isolate and improve this specific classification step.
Methods
The authors compare and combine three distinct approaches:
- Pattern Matching: Using patterns learned via distant supervision (connecting known facts in a database to text) and Universal Schema relations.
- Support Vector Machines (SVMs): They tested two versions—one using simple Bag-of-Words (BOW) and another using more sophisticated skip n-grams, which allow for "wildcards" between words to handle variations in sentence structure.
- Convolutional Neural Networks (CNNs):
- Piecewise CNN (Baseline): A model that performs convolution on the whole sentence and then splits the results for pooling.
- Contextwise CNN (Proposed): The authors' novel tweak. They split the sentence into three parts (left context, middle context, and right context) before the convolution step. This forces the filters to learn features specific to those regions without being "distracted" by the relation arguments themselves. They also use weight sharing across these regions to reduce parameters and increase robustness.
Finally, they use a Model Combination (CMB) approach, which is a simple linear interpolation of the scores from all three methods.
Datasets
- Training Data: Generated via distant supervision, matching Freebase tuples (subject, relation, object) against massive corpora including Wikipedia, the New York Times, and Clueweb.
- Evaluation Benchmark: The authors created a new, high-quality benchmark by extracting and cleaning 39,386 instances from the 2012–2014 TAC Shared Task evaluations. This is a significant contribution, as it provides a standardized way to test relation classifiers on "noisy" real-world data rather than curated datasets.
- Genres: The data is split between Newswire (87.5% of dev) and Web/Forums (12.5% of dev), allowing for an analysis of how models handle different writing styles.
Key Insights and Ideas
- The Power of Pre-Convolution Splitting: By splitting the context before convolution, the Contextwise CNN focuses on the words that actually trigger the relation. This outperformed the Piecewise CNN, jumping from an $F_1$ of 0.52 to 0.60 on the development set.
- Neural Networks Generalize Better: Unlike patterns, which require exact matches, CNNs use word embeddings. This allows them to recognize that "is based in" and "is located in" are semantically similar, even if the model only saw one of them during training.
- SVMs are Surprisingly Resilient: SVMs using skip n-grams were very competitive with CNNs. This is because skip n-grams act as a "poor man's" version of the generalization CNNs provide, allowing the model to ignore unimportant insertions between key words.
- The Genre Gap: Models trained on news data perform significantly worse on web and forum data. This highlights that "slot filling" isn't just a linguistic problem; it’s a domain adaptation problem.
Key Results
- Best Individual Model: The SVM with skip n-grams and the Contextwise CNN were neck-and-neck, with the SVM slightly leading on the evaluation set (.48 vs .46 $F_1$).
- The "Dream Team" (CMB): Combining Patterns + SVM + CNN yielded the best overall results, achieving an $F_1$ of 0.53 on the 2014 evaluation data, outperforming the traditional Mintz++ baseline (.41).
- Benchmark Validity: The authors found a high correlation (up to 0.89) between their relation classification benchmark and end-to-end system performance, proving that improving this specific component leads to better real-world systems.
Limitations and Future Work
- Domain Sensitivity: The models still struggle with the "noisy" nature of web forums. The authors suggest that future work should focus more on domain adaptation.
- Pipeline Dependency: While this paper isolates relation classification, in a real system, errors in sentence splitting or Named Entity Recognition (NER) still "leak" into the classifier, providing it with incomplete or wrong arguments.
- Linguistic Structure: The authors mention they want to explore using dependency parsing to provide the CNN with more linguistically motivated input representations, rather than just raw word sequences.
Summary updated