Combining Word Embeddings and Feature Embeddings for Fine-grained Relation Extraction

Mo Yu, Matthew R. Gormley, Mark DredzeNAACL 2015 · OpenReview

TL;DR

This paper introduces a novel method for fine-grained relation extraction by combining word embeddings with hand-crafted features more effectively. Their approach, Low-Rank FCM (LRFCM), learns feature embeddings jointly with model parameters, allowing it to scale to a large number of features and labels without overfitting, achieving state-of-the-art results.

What Problem Does This Paper Address?

Relation extraction (RE) aims to identify semantic relationships between entities in text (e.g., "Person X works for Company Y"). Modern NLP models often use word embeddings to represent words in a low-dimensional space, capturing semantic and syntactic properties. Compositional models then build representations for larger linguistic structures (like sentences) by combining these word embeddings.

While incorporating hand-crafted features (e.g., dependency path information, entity types) can significantly boost RE performance, previous methods struggled to combine them effectively. The complexity of integrating many features with word embeddings in compositional models led to models with too many parameters, risking overfitting, especially when dealing with a large number of relation types (fine-grained RE). This limited the applicability of these powerful feature-based compositional models.

Methods

The authors propose a new model called Low-Rank FCM (LRFCM). It builds upon the Factor-based Compositional Embedding Model (FCM) from Yu et al. (2014). The core idea of FCM is to use the outer product of word embeddings and feature vectors to create a representation that is then fed into a tensor-based classifier. However, this approach leads to a large tensor size, proportional to the number of features.

LRFCM addresses this by introducing a feature embedding layer. Instead of using raw feature vectors, LRFCM learns low-dimensional embeddings for each feature. These feature embeddings are then combined with word embeddings via an outer product, and this combined representation is fed into a low-rank tensor (hence "Low-Rank FCM"). This significantly reduces the number of model parameters, making the model more scalable and less prone to overfitting. The model is trained using stochastic gradient descent with AdaGrad.

Datasets

The paper evaluates LRFCM on two standard relation extraction datasets:

  • ACE 2005: A widely used dataset for relation extraction, containing both coarse and fine-grained relation types. The authors use the English portion and follow standard splits for training, development, and testing.
  • ERE (Entity Relation Extraction): A newer dataset introduced by the Linguistic Data Consortium. The authors use the third release and create their own splits for training, development, and testing.

They report results under two settings: with gold entity spans and types, and with gold entity spans but without gold entity types.

Key Insights and Ideas

  • Feature Embeddings are Key to Scalability: The most significant insight is that learning embeddings for features, rather than using them directly, is crucial for scaling compositional models to a large number of features and relation types. This dramatically reduces parameter count and combats overfitting.
  • Low-Rank Assumption for Efficiency: By assuming the interaction tensor is low-rank, the model can capture complex feature-word embedding interactions with far fewer parameters than a full tensor.
  • Bridging the Gap Between Features and Embeddings: LRFCM elegantly conjoins the strengths of hand-crafted features (expressiveness, ability to capture long-distance dependencies) with the generalization power of learned word embeddings within a unified compositional framework.
  • Fine-grained RE is a Tougher Challenge: The paper highlights that fine-grained relation extraction, with its many labels, is particularly susceptible to overfitting, making LRFCM's ability to generalize more impressive.

Key Results

  • State-of-the-Art Performance: LRFCM achieves state-of-the-art results on both ACE 2005 and ERE fine-grained relation extraction tasks.
  • Reduced Overfitting: Compared to FCM, LRFCM shows a significantly smaller gap between training and testing F1 scores, indicating less overfitting, especially in fine-grained settings. For example, on fine relations, the gap was 31.2% for LRFCM versus 40.2% for FCM.
  • Improved Generalization: LRFCM outperforms both FCM and a strong feature-rich baseline (PM'13) on most tasks, particularly when dealing with more labels.
  • Parameter Efficiency: The number of parameters in LRFCM is dramatically reduced from O(md|L|) for FCM to O(dgd|L| + dgm) for LRFCM, where d is word embedding size, dg is feature embedding size, m is number of features, and |L| is number of labels. For instance, for ACE with 32 labels and 3000 features, FCM had ~1.92x10^7 parameters, while LRFCM with 20-dim feature embeddings had ~1.28x10^5.

Limitations and Future Work

  • Complementarity of Models: The authors note that FCM and LRFCM can be complementary, suggesting that combining their predictions might further boost performance.
  • Future Work: They propose exploring combinations of FCM and LRFCM, as well as their combination with baseline methods, for future work. They also mention that while LRFCM might introduce some bias due to the low-rank approximation, it significantly reduces variance, leading to better overall generalization.

Summary updated

Abstract

Compositional embedding models build a representation for a linguistic structure based on its component word embeddings. While recent work has combined these word embeddings with hand crafted features for improved performance, it was restricted to a small number of features due to model complexity, thus limiting its applicability. We propose a new model that conjoins features and word embeddings while maintaing a small number of parameters by learning feature embeddings jointly with the parameters of a compositional model. The result is a method that can scale to more features and more labels, while avoiding overfitting. We demonstrate that our model attains state-of-the-art results on ACE and ERE fine-grained relation extraction.
BibTeX
@inproceedings{yu2015combining,
  author = {Mo Yu and Matthew R. Gormley and Mark Dredze},
  title = {{Combining Word Embeddings and Feature Embeddings for Fine-grained Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  url = {https://openreview.net/forum?id=BJ-tjXWOZr},
}
Figure