Abstract: Compositional embedding models build a representation (or embedding) for a linguistic structure based on its component word embeddings. We propose a Feature-rich Compositional Embedding Model (FCM) for relation extraction that is expressive, generalizes to new domains, and is easy-to-implement. The key idea is to combine both (unlexicalized) handcrafted features with learned word embeddings. The model is able to directly tackle the difficulties met by traditional compositional embeddings models, such as handling arbitrary types of sentence annotations and utilizing global information for composition. We test the proposed model on two relation extraction tasks, and demonstrate that our model outperforms both previous compositional models and traditional feature rich models on the ACE 2005 relation extraction task, and the SemEval 2010 relation classification task. The combination of our model and a loglinear classifier with hand-crafted features gives state-of-the-art results.
Introduction
Relation Extraction (RE) is the task of identifying semantic relationships between entities in a sentence. For example, in the sentence "A man is driving a taxi," the goal is to identify the Artifact-User relation between "man" and "taxi."
Historically, researchers faced a trade-off between hand-crafted features and word embeddings. Hand-crafted features are precise but struggle to generalize to unseen words. Conversely, word embeddings generalize well by capturing semantic similarities but often ignore the broader linguistic context or structural roles of words. The Feature-rich Compositional Embedding Model (FCM) bridges this gap by combining the generalization power of embeddings with the structural precision of linguistic features.
The FCM Architecture
The core innovation of the FCM is its ability to combine unlexicalized features with dense word embeddings using a log-bilinear approach—specifically, an outer product. This allows the model to create a "smoothed" version of traditional features where the model looks for any word with a similar embedding in a specific structural role.
The process works in three distinct stages:
- Decomposition: The sentence is broken down into substructures consisting of individual words and their associated linguistic annotations.
- Substructure Embedding: For each word $w_i$, the model takes a hand-crafted feature vector $f_{w_i}$ (e.g., binary indicators for being on a dependency path) and a dense word embedding $e_{w_i}$. It combines them using an outer product ($f_{w_i} \otimes e_{w_i}$), creating a matrix where features weight different parts of the embedding.
- Composition: These matrices are summed to form a single "annotated sentence embedding," which is then passed through a softmax layer to predict the relation label.
Experimental Evaluation
The model was evaluated on two major benchmarks: the ACE 2005 dataset for cross-domain relation extraction and the SemEval 2010 Task 8 for relation classification.
On the SemEval 2010 benchmark, the FCM achieved an F1 score of 83.4, outperforming standard RNNs and MVRNNs. On the ACE 2005 dataset, a hybrid model combining FCM with a traditional log-linear classifier achieved an average F1 score of 58.26, significantly outperforming the baseline log-linear model (54.17). This suggests that discrete features and embedding-based features capture complementary information.
Key Insights and Conclusion
The FCM demonstrates superior generalization, particularly in out-of-domain settings, by handling vocabulary shifts between different text types (e.g., news vs. weblogs). Furthermore, the model is highly efficient; unlike CNNs or RNNs, the FCM has $O(snd)$ complexity (where $s$ is feature sparsity), making it significantly faster for both training and inference.
In summary, the Feature-rich Compositional Embedding Model provides an expressive, efficient, and easy-to-implement framework for relation extraction that achieves state-of-the-art performance by effectively merging linguistic structure with semantic embeddings.
Summary updated