Adaptive Threshold Selective Self-Attention for Chinese NER

Biao Hu, Zhen Huang, Minghao Hu, Ziwen Zhang, Yong DouCOLING 2022 · OpenReview

Standard Transformer models for Chinese Named Entity Recognition (NER) often struggle with "noisy" context because their self-attention mechanism attends to every character in a sentence, regardless of relevance. This paper introduces Adaptive Threshold Selective Self-Attention (ATSSA), a data-driven mechanism that dynamically calculates a unique importance threshold for every query, filtering out irrelevant characters to sharpen entity boundaries and achieving state-of-the-art performance across four major benchmarks.

The Problem: Noisy Context in Chinese NER

In Chinese NER, identifying where an entity (like a person or organization) starts and ends is particularly difficult because Chinese lacks natural word delimiters like spaces. While Transformer-based models are excellent at capturing long-range dependencies, their fully connected self-attention is a double-edged sword.

By forcing every character to attend to every other character, the model often integrates "noise" from irrelevant parts of the sentence. For example, in the phrase "Shanghai popular culture and art festival," a standard model might attend too heavily to "popular culture," leading it to misidentify "Shanghai popular culture" as a single organization rather than identifying "Shanghai" as a location. The authors argue that for NER, a query should only focus on the most relevant keys; everything else is a distraction that blurs entity boundaries.

Comparison between Selective and Fully Connected Self-Attention mechanisms.

Proposed Method: ATSSA

The authors propose ATSSA, which replaces the "attend-to-everything" approach with a selective process. The core components include:

  1. Adaptive Threshold Generation: Instead of using a fixed sparsity pattern, the model uses a "matching matrix" to compare a specific character against the entire sequence. It then generates a unique threshold ($T_i$) for each query character.
  2. Differentiable Binarization: To make the selection process trainable, the authors use an approximate step function and Bernoulli sampling (via the Gumbel-Softmax trick). This allows the model to "decide" which keys to keep and which to discard while still being able to pass gradients during backpropagation.
  3. Top-k Constraint: To ensure the model doesn't accidentally discard too much information during early training, they implement a safety net that forces the model to keep at least the top $k$ most relevant keys (where $k=3$ was found to be optimal).
  4. Integration with FLAT: The mechanism is built on top of the FLAT (Flat-Lattice Transformer) architecture, which incorporates word-level lexicon information into character-level embeddings.
Detailed architecture of the Adaptive Threshold Selective Self-Attention mechanism.

Key Insights

  • Sparsity for Denoising: Unlike many sparse attention papers that aim for computational speedups, ATSSA uses sparsity as a tool for denoising. By ignoring irrelevant characters, the model produces a "sharper" attention map that better respects entity boundaries.
  • Dynamic Focus: Previous attempts at sparse attention used fixed patterns (like attending only to neighbors). ATSSA is data-driven; the focus of the attention changes based on the specific meaning of the sentence.
  • Faster Convergence: Interestingly, while the selective mechanism adds a tiny bit of overhead, the model actually converges faster than the baseline FLAT model because it isn't distracted by irrelevant character interactions.

Experimental Results

The model was evaluated on four standard benchmark datasets: Weibo NER, Resume NER, OntoNotes 4.0, and MSRA.

  • Performance Boost: ATSSA achieved an average improvement of 1.68 F1 points over the baseline FLAT model across all datasets.
  • State-of-the-Art: The model reached SOTA results on all four benchmarks, notably hitting 72.53 F1 on Weibo and 96.73 F1 on Resume.
  • Efficiency: Despite the extra thresholding math, the model is only about 5% slower in inference speed than the standard FLAT model, making it highly practical for real-world deployment.
  • Qualitative Success: Case studies showed that ATSSA correctly separated adjacent entities (like "Tencent" and "Lenovo") that standard Transformers often merged into a single, incorrect entity.

Conclusion

ATSSA demonstrates that selective attention is a powerful tool for improving the precision of Chinese NER. By dynamically filtering out noise, the model achieves superior boundary detection with minimal impact on inference speed. Future work may involve adapting this adaptive thresholding mechanism to other NLP tasks like text classification and natural language inference.

Summary updated

BibTeX
@inproceedings{hu2022adaptive,
  author = {Biao Hu and Zhen Huang and Minghao Hu and Ziwen Zhang and Yong Dou},
  title = {{Adaptive Threshold Selective Self-Attention for Chinese NER}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=SEek-HW7tH},
}
Figure