TL;DR
This paper demonstrates that Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) capture complementary information for relation classification without relying on manual linguistic features. By introducing an "extended middle context" for CNNs, a novel connectionist bi-directional architecture for RNNs, and a pairwise ranking loss, the authors combine both model families via majority voting to achieve a new state of the art on the SemEval-2010 benchmark (84.9 Macro F1).
What Problem Does This Paper Address?
Relation classification involves identifying the semantic relationship between two tagged entities within a sentence (e.g., determining that "We poured the <e1>milk</e1> into the <e2>pumpkin mixture</e2>" expresses Entity-Destination(e1, e2)).
Prior approaches leaned heavily on handcrafted linguistic resources, dependency parsers, or isolated neural architectures (either standalone CNNs or recursive/recurrent networks). Standalone CNNs often lose focus when sentence length grows—pooling irrelevant words far from the target entities—while traditional RNNs struggle to carry early-sentence signals all the way to the final decision. This work asks: How can we optimize each architecture’s structural representation, and can combining their distinct inductive biases yield stronger semantic understanding?
Methods
The authors introduce architectural refinements for both model families and unify them through an ensemble:
- CNN with Extended Middle Context: Rather than feeding the entire sentence or solely the text between entities, the input is split into two overlapping segments:
- Left context + Left entity + Middle context
-
Middle context + Right entity + Right context
Each branch undergoes independent convolution and max-pooling before being concatenated. Repeating the middle context prevents max-pooling from being hijacked by irrelevant tokens in long sentences while still preserving outer context. -
Connectionist Bi-directional RNN (C-BRNN):
- Uses concatenated word trigrams as inputs instead of individual word tokens.
-
Merges standard forward ($\vec{h}_t$) and backward ($\overleftarrow{h}_t$) hidden states with an additional recurrent connection to the previous combined hidden state ($h_{t-1}$) via weight matrix $H$: $$h_t = f(\overleftarrow{h}_t + \vec{h}_t + H \cdot h_{t-1})$$ This allows intermediate representations across all time steps to directly influence the final sentence-level classification.
-
Ranking Loss: Instead of traditional cross-entropy with softmax, the authors optimize both CNNs and RNNs with a margin-based ranking objective that maximizes the gap between the true label and the highest-scoring competing label, explicitly down-weighting the negative/artificial class
"Other". - Ensemble by Voting: Predictions from multiple trained CNN and RNN models are combined via a simple majority vote (with ties broken randomly among the most frequent classes).
Datasets
- SemEval-2010 Task 8: The standard benchmark for semantic relation extraction between nominals, containing 8,000 training sentences and 2,717 test sentences. It includes 19 relation classes (9 directed relations with two directions each, plus a catch-all
"Other"category). Evaluated using the official macro-averaged F1 score. - Pretrained Word Embeddings: 50-dimensional and 400-dimensional embeddings trained via
word2vecon an English Wikipedia dump (May 2014, ~6.7 billion tokens).
Key Insights and Ideas
- Complementary Inductive Biases: CNNs act as local n-gram feature selectors (extracting peak activations via max-pooling), while RNNs build continuous, sequence-weighted semantic trajectories. Ensembling them outperforms scaling up either model alone.
- Ranking Loss Transcends Architecture: While previously proven effective on CNNs, this paper demonstrates that margin-based ranking loss provides an enormous boost to RNNs as well (+3.0 F1 jump over softmax).
- Context Repetition as Attention: Duplicating the "middle context" functions as a lightweight, structural pseudo-attention mechanism, dramatically mitigating max-pooling degradation on longer sequences without complex attention layers.
Key Results
All reported metrics use the official SemEval-2010 Task 8 Macro F1 score:
- CNN Ablations:
- Baseline (50d embeddings, middle context only): 73.0
- Adding position embeddings: 78.6
- Adding multi-window filters (sizes 2–5): 78.7
- Replacing softmax with ranking loss: 81.9
- Adding extended middle context: 82.2
- Scaling embeddings to 400d: 83.9 (Single model) / 84.2 (CNN ensemble).
- RNN Ablations:
- Baseline unidirectional RNN: 61.2
- Adding position indicators: 73.4
- Bidirectional RNN: 74.2
- Connectionist Bi-RNN: 78.4 (+4.2 boost from hidden-to-hidden bridge)
- Adding ranking loss & 400d embeddings: 82.5 (Single model) / 83.4 (RNN ensemble).
- Overall State of the Art:
- Prior SOTA (CR-CNN, Dos Santos et al., 2015): 84.1
- Proposed ER-CNN + R-RNN Ensemble: 84.9 (achieved completely free of syntactic parses or hand-crafted linguistic features).
Limitations and Future Work
- Naive Combination Strategy: The networks are merged using simple post-hoc majority voting rather than an end-to-end, jointly trained hybrid model (e.g., feeding CNN features directly into an RNN or a shared dense layer).
- Tie-Breaking Heuristic: Ties in the voting ensemble are resolved randomly among frequent classes, which introduces slight non-determinism.
- Evaluation Breadth: Experiments were confined to a single benchmark (SemEval-2010 Task 8), leaving performance on cross-sentence or larger-scale distantly supervised relation extraction unexplored.
Summary updated