A Label-Aware Autoregressive Framework for Cross-Domain NER

Jinpeng Hu, He Zhao, Dan Guo, Xiang Wan, Tsung-Hui ChangNAACL 2022 · DOI

Introduction

Named Entity Recognition (NER) is a fundamental task in natural language processing, yet models often struggle when transitioned from a data-rich source domain (e.g., general news) to a specialized target domain (e.g., scientific literature). Traditional approaches focus heavily on aligning token representations across domains but frequently overlook the semantic information inherent in the labels themselves. This paper, presented at NAACL 2022, introduces a novel Label-Aware Autoregressive Framework (LANER) that leverages label semantics and long-range dependencies to bridge the domain gap.

The Challenge of Cross-Domain NER

Standard NER models typically rely on Conditional Random Fields (CRF) to model label dependencies. However, CRFs are limited to local, adjacent label constraints, failing to capture broader document-level context. Furthermore, existing cross-domain methods often treat labels as arbitrary indices rather than meaningful concepts. For instance, the word "Bayes" might be a Person in a news context but an Algorithm in AI research. Without understanding the semantic nature of these labels and their historical context within a sequence, models struggle with such ambiguities.

Methodology: The LANER Framework

The proposed framework treats NER as an autoregressive task, where the model "remembers" previously assigned labels to inform current predictions.

Architecture of the Label-Aware Autoregressive Framework, highlighting the Bi-Attention and Label Embedding components.

The architecture consists of three primary modules:

  1. Input Sequence Encoder: Utilizes a pre-trained BERT model to generate contextualized token representations ($h_i$).
  2. Label Encoder: Assigns trainable embeddings to each possible label. A Bi-LSTM then encodes the sequence of previously assigned labels, capturing the "internal context" of the label sequence.
  3. Label Predictor (Bi-Attention Module): A dual-attention mechanism fuses text and label features. The "Label Background" component uses label history to query the input sentence, while "Label Context" uses token features to query the label history.

This design allows the model to dynamically weigh textual evidence against label-sequence logic, which is particularly effective when dealing with domain-specific terminology.

Experimental Results

The authors evaluated LANER using CoNLL2003 as the source domain and several CrossNER datasets (Politics, Science, Music, Literature, AI) as targets, alongside MIT Movie and Restaurant datasets.

Performance comparison of LANER against various baselines across multiple target domains.

As shown in the results, LANER consistently outperforms existing baselines. In the AI domain, it achieved an F1 score of 61.72, a significant improvement over the 58.28 achieved by the Multi-Cell LSTM. When integrated with Domain Adaptive Pre-training (DAPT), the model reached an impressive average F1 of 72.31 across all CrossNER domains. Notably, these gains are achieved with minimal overhead, adding only approximately 1% more parameters compared to a standard BERT-based tagger.

Key Insights and Limitations

The success of LANER highlights that labels possess semantic value that can be transferred across domains. By treating labels as concepts rather than integers, the model recognizes that a "Person" in news shares characteristics with a "Researcher" in AI.

However, the framework does face limitations. The performance gains are most pronounced in cross-domain scenarios where text is ambiguous; in-domain improvements are less dramatic. Additionally, the autoregressive nature of the model—predicting labels sequentially—results in slower inference speeds compared to parallel sequence taggers. Future research may focus on parallelizing this process or addressing source-domain biases where the model might misclassify fine-grained entities based on source-domain tendencies.

Conclusion

By explicitly modeling label semantics and long-range dependencies through an autoregressive approach, the Label-Aware Autoregressive Framework sets a new benchmark for cross-domain NER. It demonstrates that the "other half" of the data—the labels—is just as critical as token representations for effective domain adaptation.

Summary updated

BibTeX
@inproceedings{hu2022label,
  author = {Jinpeng Hu and He Zhao and Dan Guo and Xiang Wan and Tsung-Hui Chang},
  title = {{A Label-Aware Autoregressive Framework for Cross-Domain NER}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.171},
}
Figure