TL;DR
This paper introduces type-aware convolutional neural networks (CNNs) for the relation classification step in Knowledge Base Population (KBP) slot filling pipelines. By explicitly modeling entity types—via pipeline features, multi-task joint training, or linear-chain CRF structured prediction—the authors reduce error propagation and achieve state-of-the-art slot filling results among systems trained on distantly supervised data.
What Problem Does This Paper Address?
In the TAC KBP slot filling task, a system answers relational queries about entities (e.g., finding the founder of "Apple" or the spouse of a person) by reading millions of unstructured documents. Standard systems rely on multi-stage pipelines: information retrieval, candidate extraction, entity linking, and relation classification.
This setup presents two major hurdles for relation classifiers:
- Severe Label and Input Noise: Training sets rely on noisy distant supervision, while test-time inputs suffer from upstream pipeline failures (e.g., incorrect named-entity recognition, poor sentence segmentation, coreference errors).
- Entity Overfitting vs. Semantic Ambiguity: Using raw entity tokens causes neural models to overfit to specific entity names in the training set. However, completely removing entities destroys critical semantic context (e.g., distinguishing between "Apple launches iPhone" [product launch] and "SpaceX launches Falcon 9" [spaceflight]).
Methods
The authors build on a segment-based CNN architecture. Sentences are split into three parts: left, middle (between entity and filler candidate), and right context. Each segment passes through shared convolutional filters followed by 3-max pooling, combined with an argument-order indicator flag.
To supply entity semantics without causing lexical overfitting, the authors evaluate three type-aware variations:
- Pipeline-based: Appends coarse-grained one-hot type vectors (Person, Organization, Location, Date, Number, Other) to the sentence representation via learned type embeddings.
- Joint Training: Simultaneously trains an auxiliary CNN to predict argument entity types (using multi-label sigmoid loss) and the main relation classifier via a combined multi-task loss.
- Structured Prediction: Treats relation classification and entity typing as an interdependent sequence:
[class(e1), relation(e1, e2), class(e2)]. Scores from CNN representations are fed into a linear-chain Conditional Random Field (CRF) that models transitions between entity classes and relation types.
Datasets
- Distant Supervision Training Corpus: Distantly labeled examples gathered by pairing Freebase with the TAC source corpus (LDC2013E45), NYT corpus, ClueWeb subset, and Wikipedia, then filtered using high-precision patterns and self-trained SVMs.
- Slot Filling Benchmark (Dev/Test): Constructed from manual system assessment logs of TAC KBP 2012–2013 (dev) and 2014 (test) for standalone relation classification.
- Evaluation Benchmark: TAC KBP 2015 Cold Start Slot Filling evaluation data (1,951 queries, including 930 two-hop queries).
- Released Coreference Resource: Stanford CoreNLP annotations over the entire 2-million-document TAC corpus (198M mentions across 54M coreference chains).
Key Insights and Ideas
- Mutual Entity-Relation Constraints Help Precision: While standalone binary classifiers outperformed multi-class CNNs on isolated benchmarks, multi-class type-aware CNNs won inside the end-to-end pipeline. The 1-out-of-$k$ softmax constraint, combined with type compatibility, dramatically suppressed false-positive predictions.
- Joint and Structured Modeling Mitigate Upstream Errors: In downstream slot filling, upstream NER frequently misidentifies entity types. Both joint training and CRF structured prediction allow relation context to "correct" erroneous entity types.
- Bottlenecks Lie in Downstream Filtering: The authors' recall analysis revealed that even with a perfect relation classifier, pipeline recall would max out at 59.64% (theoretical max $F_1$ of 74.72%). Furthermore, error analysis showed that 61% of pipeline false positives stemmed directly from the relation classification module—predominantly when encountering long sentences mentioning multiple entities.
Key Results
- TAC KBP 2015 Slot Filling Performance: The joint type-aware CNN achieved 22.17% overall micro $F_1$ (27.97% P / 18.36% R), ranking at the top of all distantly supervised slot filling systems submitted to the benchmark.
- Macro $F_1$ Gains on Rare Slots: The structured prediction (CRF) model dominated macro $F_1$ metrics, reaching 25.12% overall macro $F_1$ and 9.24% on Hop 1 (compared to 6.13% for the baseline binary SVM).
- Ablation Studies:
- Removing the CNN dropped overall pipeline micro $F_1$ by 10.68 points (from 22.17% down to 11.49%).
- Omitting coreference resolution dropped $F_1$ by 1.32 points (and reduced absolute true positives from 398 to 366).
- Removing entity linking caused a modest drop of 0.49 $F_1$ points, indicating that entity ambiguity is a minor source of error compared to candidate classification.
Limitations and Future Work
- Training Label Noise: Distantly supervised annotations still contain considerable false positives and false negatives, lagging behind fully supervised, crowdsourced baselines (e.g., Stanford's 31.06% micro $F_1$).
- Long-Distance Dependencies: Flat convolutions struggled on long contexts containing multiple interacting entities (e.g., confusing birthplace vs. death place when several cities appear in one sentence).
- Future Directions: Incorporating attention mechanisms and syntactic dependency structures into joint models, as well as developing semi-supervised denoising methods that reduce reliance on pure distant supervision.
Summary updated