Reliability-aware Dynamic Feature Composition for Name Tagging

Ying Lin, Liyuan Liu, Heng Ji, Dong Yu, Jiawei Han

Abstract

Standard name tagging models often struggle with rare or unseen entities because they treat all word embeddings as equally reliable. This paper introduces a reliability-aware gating mechanism that uses word frequency signals to dynamically balance word embeddings with character-level and contextual features. Evaluated on the OntoNotes 5.0 corpus, the proposed model shows significant performance gains, particularly in cross-genre scenarios where entity overlap is minimal.

Introduction

Name tagging, a fundamental task in Named Entity Recognition (NER), involves identifying and classifying entities such as people, organizations, and locations. Modern neural models rely heavily on pre-trained word embeddings (e.g., GloVe). However, these embeddings are subject to the "long tail" distribution: while common words have high-quality vectors, rare words—which names often are—result in noisy and unreliable representations. Current models are typically "reliability-agnostic," consuming all vectors with equal weight regardless of their quality.

Methodology: Reliability-Aware Gating

The authors propose a model that utilizes Reliability Signals to guide a dynamic feature composition process at two distinct levels. These signals are derived from word frequencies in both the pre-training corpus and the task-specific training set, serving as a proxy for embedding trustworthiness.

Overview of the reliability-aware architecture, illustrating how reliability signals drive gating at the word representation and feature extraction levels.

Word Representation Level Gating

Instead of simple concatenation, the model uses a gate to weigh word embeddings against character-level representations. If a word is rare (e.g., "MedChem"), the gate suppresses the noisy word vector and prioritizes character-level clues like prefixes or suffixes. This is further enhanced by splitting the character representation into semantic and morphological components.

Feature Extraction Level Gating

A key innovation is the introduction of "context-only" features—representations derived strictly from surrounding words, excluding the target word itself. A second set of gates balances the standard LSTM hidden state with these context-only features. This forces the model to leverage linguistic environments (e.g., "a small town like...") when the target word is unknown.

Detailed view of the feature extraction level gating, showing how context reliability signals help the model prioritize surrounding text for unknown words.

Experimental Results

The model was evaluated on the OntoNotes 5.0 corpus across six diverse genres, including Newswire, Telephone Conversations, and Weblogs.

  • In-Genre Gains: The model outperformed the state-of-the-art LSTM-CNN-CRF baseline across all genres, with F-score improvements of up to 6.2% in the Telephone Conversation genre.
  • Cross-Genre Robustness: In cross-genre experiments (training on one genre and testing on another), the model achieved an average F-score gain of 2.3%, with improvements as high as 9.6% in the most challenging cases.
  • Handling Unseen Entities: Qualitative analysis demonstrated that the model effectively identifies names it has never seen by correctly identifying when context is more reliable than an unknown word's vector.

Conclusion

By making neural models "aware" of embedding reliability through frequency-based signals, this work provides a robust fallback mechanism for handling the long tail of language. While the current implementation uses LSTMs, the gating logic is complementary to modern Transformer-based architectures. Future research may explore more sophisticated reliability signals and apply this dynamic composition to tasks like event extraction.

Summary updated

BibTeX
@inproceedings{lin2019reliability,
  author = {Ying Lin and Liyuan Liu and Heng Ji and Dong Yu and Jiawei Han},
  title = {{Reliability-aware Dynamic Feature Composition for Name Tagging}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1016},
}
Figure