Dynamic Modeling Cross- and Self-Lattice Attention Network for Chinese NER

Shan Zhao, Minghao Hu, Zhiping Cai, Haiwen Chen, Fang LiuAAAI 2021 · OpenReview

Abstract

Word-character lattice models have been proved to be effective for Chinese named entity recognition (NER), in which word boundary information is fused into character sequences for enhancing character representations. However, prior approaches have only used simple methods such as feature concatenation or position encoding to integrate word-character lattice information, but fail to capture fine-grained correlations in word-character spaces. In this paper, we propose DCSAN, a Dynamic Cross- and Self-lattice Attention Network that aims to model dense interactions over word-character lattice structure for Chinese NER. By carefully combining cross-lattice and self-lattice attention modules with gated word-character semantic fusion unit, the network can explicitly capture fine-grained correlations across different spaces (e.g., word-to-character and character-to-character), thus significantly improving model performance. Experiments on four Chinese NER datasets show that DCSAN obtains state-of-the-art results as well as efficiency compared to several competitive approaches.

Introduction

Named Entity Recognition (NER) in Chinese is significantly more challenging than in English due to the lack of explicit word boundaries. Traditional approaches often rely on a two-step pipeline: word segmentation followed by entity labeling. However, this method suffers from error propagation, where segmentation mistakes negatively impact downstream NER performance.

To mitigate this, "lattice" models were developed to integrate word-level information directly into character-based models. While effective, early iterations like Lattice LSTM relied on simple feature concatenation or position encodings, which failed to capture fine-grained correlations between characters and potential word matches. Furthermore, sequential RNN-based architectures are inherently slow and difficult to parallelize.

Comparison of different lattice-based modeling strategies: (a) Original Lattice LSTM, (b) Soft-lexicon feature strategy, and (c) the proposed dense interactions of lattice inputs.

Methodology

The proposed DCSAN (Dynamic Cross- and Self-lattice Attention Network) treats characters and words as distinct "modalities," borrowing concepts from multimodal learning in computer vision to capture dense interactions.

The overall architecture of DCSAN, featuring the Cross-lattice Attention Module, Gated Fusion, and Dynamic Self-lattice Attention Module.

The architecture consists of four primary components:

  1. Soft-Lexicon Representation: Instead of complex graphs, word information is represented using a "soft-lexicon" strategy. Matched words for each character are categorized into four sets (Begin, Middle, End, Single) and condensed into fixed-dimensional vectors.
  2. Cross-Lattice Attention Module: This module treats words as "queries" and characters as "keys/values," forcing the model to learn pairwise relationships and creating "word-aware" character representations.
  3. Gated Fusion Unit: A sigmoid-based gate dynamically determines the optimal balance between character and word features for each position, filtering out irrelevant word information.
  4. Dynamic Self-Lattice Attention Module: Utilizing self-attention with relative position encoding, this module captures long-range dependencies between characters, essential for identifying long entity names.

The final output is processed through a Conditional Random Field (CRF) layer to ensure globally optimal label sequences.

Experiments and Results

DCSAN was evaluated on four standard Chinese NER benchmarks: Resume (professional), MSRA (news), Weibo (social media), and E-commerce.

Key Findings

  • Performance: DCSAN achieved state-of-the-art accuracy across all datasets. Most notably, it reached an F1 score of 71.27% on Weibo, a substantial improvement over the previous best of 68.55%.
  • Efficiency: By replacing sequential LSTMs with parallelizable attention modules, DCSAN demonstrated significant speedups. Inference on the Weibo dataset was 6.64 times faster than the classic Lattice LSTM.
  • Ablation Study: Removing the cross-lattice attention module resulted in a 1.66-point drop in F1 score, confirming that dense word-to-character interactions are vital for performance.

Conclusion

DCSAN introduces a novel perspective by viewing Chinese NER as a multimodal interaction problem between characters and words. By utilizing cascaded cross- and self-lattice attention, the model captures fine-grained correlations while maintaining high computational efficiency. Future work may explore the integration of diverse pre-trained language models like RoBERTa and the application of this cross-modal approach to other logographic languages like Japanese.

Summary updated

Abstract

Word-character lattice models have been proved to be effective for Chinese named entity recognition (NER), in which word boundary information is fused into character sequences for enhancing character representations. However, prior approaches have only used simple methods such as feature concatenation or position encoding to integrate word-character lattice information, but fail to capture fine-grained correlations in word-character spaces. In this paper, we propose DCSAN, a Dynamic Cross- and Self-lattice Attention Network that aims to model dense interactions over word-character lattice structure for Chinese NER. By carefully combining cross-lattice and self-lattice attention modules with gated word-character semantic fusion unit, the network can explicitly capture fine-grained correlations across different spaces (e.g., word-to-character and character-to-character), thus significantly improving model performance. Experiments on four Chinese NER datasets show that DCSAN obtains stateof-the-art results as well as efficiency compared to several competitive approaches.
BibTeX
@inproceedings{zhao2021dynamic,
  author = {Shan Zhao and Minghao Hu and Zhiping Cai and Haiwen Chen and Fang Liu},
  title = {{Dynamic Modeling Cross- and Self-Lattice Attention Network for Chinese NER}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=5Zdt0AziuVF},
}
Figure