Toward Mention Detection Robustness with Recurrent Neural Networks

Thien Huu Nguyen, Avirup Sil, Georgiana Dinu, Radu FlorianarXiv 2016 · OpenReview

Abstract

One of the key challenges in natural language processing (NLP) is to yield good performance across application domains and languages. In this work, we investigate the robustness of the mention detection systems, one of the fundamental tasks in information extraction, via recurrent neural networks (RNNs). The advantage of RNNs over the traditional approaches is their capacity to capture long ranges of context and implicitly adapt the word embeddings, trained on a large corpus, into a task-specific word representation, but still preserve the original semantic generalization to be helpful across domains. Our systematic evaluation for RNN architectures demonstrates that RNNs not only outperform the best reported systems (up to 9% relative error reduction) in the general setting but also achieve the state-of-the-art performance in the cross-domain setting for English. Regarding other languages, RNNs are significantly better than the traditional methods on the similar task of named entity recognition for Dutch (up to 22% relative error reduction).

Introduction

Mention Detection (MD) is a fundamental task in information extraction, involving the identification and classification of entity mentions—such as names ("France"), nominals ("the president"), or pronouns ("they")—into categories like PERSON or ORGANIZATION. Traditional MD systems, typically based on graphical models like Conditional Random Fields (CRFs) or Hidden Markov Models, face significant hurdles: domain shift between training and test data, limited context windows for long-range dependencies, and the high manual effort required for language portability. This paper investigates how Recurrent Neural Networks (RNNs) can overcome these limitations through distributed representations and sequence modeling.

Methodology

The researchers systematically evaluated several RNN architectures to determine the optimal configuration for robust mention detection.

Basic Architectures

The study first compared foundational models: the Elman network (hidden-to-hidden connections) and the Jordan network (output-to-hidden connections). These serve as the baseline for sequential processing in the task.

Comparison of Elman and Jordan RNN architectures.

Bidirectional and Contextual Models

To capture context from both preceding and succeeding words, bidirectional RNNs were employed. This architecture uses three passes: a forward RNN, a backward RNN, and a decoding RNN that integrates both streams to make final predictions. The authors also tested a "Contextual" model, which uses a different aggregation strategy for context.

Comparison of the Bidirectional (left) and Contextual (right) RNN architectures.

Gated Units and C-CONCAT Embeddings

Gated Recurrent Units (GRUs) were integrated to address the vanishing gradient problem, enabling the capture of long-range dependencies. Furthermore, the authors introduced C-CONCAT embeddings, a variation of the Word2Vec CBOW model where context word vectors are concatenated rather than averaged. This approach better reflects the decoder's local context during inference.

Experimental Setup

The models were benchmarked on several standard datasets:

  • ACE 2005: Used to evaluate domain robustness across six diverse domains (e.g., weblogs, telephone conversations).
  • CoNLL 2003 (English): Used for benchmarking against state-of-the-art Named Entity Recognition (NER) systems.
  • CoNLL 2002 (Dutch): Used to assess cross-lingual portability.

Word embeddings were pre-trained on the English Gigaword corpus (6 billion tokens) and Dutch Wikipedia to ensure broad semantic coverage.

Key Results and Insights

  • Bidirectional Superiority: The bidirectional architecture was the most successful, achieving an F1 score of 82.7 on ACE 2005. The study found that bidirectional modeling is essential for disambiguating mentions like "Liverpool," where the following context determines if it is a city or a sports team.
  • Cross-Domain Robustness: In feature-light settings, the bidirectional RNN significantly outperformed traditional Maximum Entropy Markov Models (MEMM), showing a 4.5% absolute F1 improvement on informal weblog data.
  • Cross-Lingual Performance: The system achieved an F1 of 83.45 on Dutch NER, representing a 22% relative error reduction over previous best results, proving the model's effectiveness across languages without extensive feature engineering.
  • Decoding Matters: Utilizing an RNN for the final decoding step helped capture dependencies between sequential labels (e.g., ensuring a "Beginning-Person" tag is followed by an "Inside-Person" tag).

Conclusion

This work demonstrates that bidirectional RNNs with recurrent decoding provide a powerful and robust framework for mention detection. By relying on dense word embeddings and gated sequence modeling, these networks generalize better across domains and languages than traditional hand-engineered feature sets, setting new benchmarks for information extraction tasks.

Summary updated

Abstract

One of the key challenges in natural language processing (NLP) is to yield good performance across application domains and languages. In this work, we investigate the robustness of the mention detection systems, one of the fundamental tasks in information extraction, via recurrent neural networks (RNNs). The advantage of RNNs over the traditional approaches is their capacity to capture long ranges of context and implicitly adapt the word embeddings, trained on a large corpus, into a task-specific word representation, but still preserve the original semantic generalization to be helpful across domains. Our systematic evaluation for RNN architectures demonstrates that RNNs not only outperform the best reported systems (up to 9\% relative error reduction) in the general setting but also achieve the state-of-the-art performance in the cross-domain setting for English. Regarding other languages, RNNs are significantly better than the traditional methods on the similar task of named entity recognition for Dutch (up to 22\% relative error reduction).
BibTeX
@misc{nguyen2016toward,
  author = {Thien Huu Nguyen and Avirup Sil and Georgiana Dinu and Radu Florian},
  title = {{Toward Mention Detection Robustness with Recurrent Neural Networks}},
  year = {2016},
  eprint = {1602.07749},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1602.07749},
}
Figure