Improved Relation Extraction with Feature-rich Compositional Embedding Models

Matthew R. Gormley, Mo Yu, Mark Dredze

TL;DR

This paper introduces the Feature-rich Compositional Embedding Model (FCM), a novel approach to relation extraction that effectively combines hand-crafted linguistic features with learned word embeddings. FCM achieves state-of-the-art results on standard benchmarks by better capturing the nuances of linguistic context and structure.

What Problem Does This Paper Address?

Relation extraction, a core task in Natural Language Processing (NLP), aims to identify semantic relationships between entities mentioned in text (e.g., "Person X works for Company Y"). Traditional methods often rely on extensive hand-crafted features, which are brittle and difficult to generalize to new domains or unseen words. While word embeddings (like Word2Vec) have shown promise in capturing semantic meaning and generalizing, they often struggle to incorporate the rich contextual and structural information crucial for relation extraction. Existing compositional embedding models, which aim to build sentence representations from word embeddings, face challenges in handling diverse linguistic annotations and global sentence context, limiting their effectiveness. This paper addresses the need for a model that can seamlessly integrate both rich linguistic features and powerful word embeddings to achieve robust and generalizable relation extraction.

Methods

The core innovation is the Feature-rich Compositional Embedding Model (FCM). FCM works by:

  1. Decomposing sentences into substructures: For relation extraction, each word in a sentence is treated as a substructure.
  2. Creating substructure embeddings: For each word (substructure), FCM combines its learned word embedding ($e_{wi}$) with a hand-crafted feature vector ($f_{wi}$) representing its local context and annotations (like dependency path information). This combination is achieved through an outer product, resulting in a matrix $h_{wi} = f_{wi} e_{wi}^T$. This matrix captures both the semantic meaning of the word and its specific role in the sentence.
  3. Forming an annotated sentence embedding: The substructure embeddings are summed to create a single embedding for the entire annotated sentence: $E = \sum_{i=1}^{n} h_{wi}$. This aggregated embedding represents the sentence's overall meaning and structure.
  4. Scoring and predicting relations: This sentence embedding is then used by a log-bilinear model (or a log-linear model if word embeddings are fixed) to predict the relation type between two entities.

The key novelty lies in the outer product combination of features and embeddings and the summation of these substructure embeddings. This allows FCM to:

  • Handle arbitrary linguistic annotations: The feature vector $f_{wi}$ can incorporate diverse information (POS tags, dependency paths, entity types, etc.).
  • Utilize global information: The summation of substructure embeddings allows for the integration of context from across the entire sentence.
  • Generalize effectively: By combining learned embeddings with structured features, FCM can generalize to unseen words and contexts.

The paper also proposes a Hybrid Model that combines FCM with a traditional log-linear model, further boosting performance.

Datasets

The paper evaluates FCM on two standard relation extraction benchmarks:

  • ACE 2005: This dataset contains relation extraction instances from six different domains (Newswire, Broadcast Conversation, Broadcast News, Telephone Speech, Usenet Newsgroups, and Weblogs). The evaluation focuses on the domain adaptation setting, where models are trained on some domains and tested on unseen ones.
  • SemEval 2010 Task 8: This dataset focuses on relation classification between entity pairs in sentences, with a reduced number of negative examples compared to ACE.

The authors also mention using pre-trained word embeddings from the English Wikipedia (w2v-enwiki-d300) and NYT portion of the Gigaword 5.0 corpus (w2v-nyt-d200).

Key Insights and Ideas

  • The power of combining features and embeddings: The paper strongly demonstrates that simply using word embeddings is insufficient. The true power comes from their integration with rich, hand-crafted linguistic features, allowing the model to understand both what a word means and how it's used in context.
  • Outer product as a "smoothed" lexical feature: The outer product $f_{wi} e_{wi}^T$ can be seen as a generalization of traditional conjunctive features (e.g., "word is X AND word is on dependency path"). By using embeddings, FCM smooths these features, allowing generalization to low-frequency words.
  • Summation of substructure embeddings is crucial: Unlike models that might concatenate or average embeddings, summing them allows FCM to build a more holistic sentence representation that captures the interplay of different linguistic elements.
  • Generalizability is key: FCM's design inherently promotes generalization, making it effective in out-of-domain settings and for unseen words, a significant advantage over purely feature-based methods.
  • Task-specific embeddings can further boost performance: While general-purpose embeddings work well, using embeddings specifically trained for the task (as explored in the SemEval experiments) can lead to even better results.

Key Results

  • State-of-the-art performance: FCM, especially when combined with a log-linear classifier, achieves state-of-the-art results on the ACE 2005 relation extraction task in the cross-domain setting.
  • Competitive results on SemEval 2010 Task 8: FCM performs competitively with other leading compositional models on this task.
  • Outperforms baselines: FCM consistently outperforms traditional log-linear models and previous compositional embedding models on both datasets.
  • Importance of features: Ablation studies show that removing specific feature sets (like "HeadEmb" or "In-between") significantly degrades performance, highlighting the value of the rich feature engineering.
  • Efficiency: FCM has a favorable time complexity for inference compared to CNNs and RNNs.

Limitations and Future Work

  • Limitations:
    • The paper notes that fine-tuning embeddings on the ACE development set did not improve performance on unseen test domains, suggesting a risk of overfitting.
  • In the low-resource setting (unknown entity types), FCM's performance drops significantly, indicating a reliance on entity type information.
  • The paper acknowledges that while FCM is efficient, further improvements could be made by incorporating task-specific enhancements like ranking-based loss functions.
  • Future Work:
    • Investigate task-specific embeddings and loss functions for further performance gains.
  • Apply FCM to other NLP tasks like dependency parsing, semantic role labeling, and paraphrasing.
  • Explore its application to tasks like TAC-KBP by adapting the training objective.
  • Further investigate the benefits of combining multiple word representations.

Summary updated

Abstract

Compositional embedding models build a representation (or embedding) for a linguistic structure based on its component word embeddings. We propose a Feature-rich Compositional Embedding Model (FCM) for relation extraction that is expressive, generalizes to new domains, and is easy-to-implement. The key idea is to combine both (unlexicalized) hand-crafted features with learned word embeddings. The model is able to directly tackle the difficulties met by traditional compositional embeddings models, such as handling arbitrary types of sentence annotations and utilizing global information for composition. We test the proposed model on two relation extraction tasks, and demonstrate that our model outperforms both previous compositional models and traditional feature rich models on the ACE 2005 relation extraction task, and the SemEval 2010 relation classification task. The combination of our model and a log-linear classifier with hand-crafted features gives state-of-the-art results.
BibTeX
@inproceedings{gormley2015improved,
  author = {Matthew R. Gormley and Mo Yu and Mark Dredze},
  title = {{Improved Relation Extraction with Feature-rich Compositional Embedding Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=B1Wr4MG_-S},
}

Loading related content...

Figure