Standard Transformer models for Chinese Named Entity Recognition (NER) often struggle with "noisy" context because their self-attention mechanism attends to every character in a sentence, regardless of relevance. This paper introduces Adaptive Threshold Selective Self-Attention (ATSSA), a data-driven mechanism that dynamically calculates a unique importance threshold for every query, filtering out irrelevant characters to sharpen entity boundaries and achieving state-of-the-art performance across four major benchmarks.
The Problem: Noisy Context in Chinese NER
In Chinese NER, identifying where an entity (like a person or organization) starts and ends is particularly difficult because Chinese lacks natural word delimiters like spaces. While Transformer-based models are excellent at capturing long-range dependencies, their fully connected self-attention is a double-edged sword.
By forcing every character to attend to every other character, the model often integrates "noise" from irrelevant parts of the sentence. For example, in the phrase "Shanghai popular culture and art festival," a standard model might attend too heavily to "popular culture," leading it to misidentify "Shanghai popular culture" as a single organization rather than identifying "Shanghai" as a location. The authors argue that for NER, a query should only focus on the most relevant keys; everything else is a distraction that blurs entity boundaries.
Proposed Method: ATSSA
The authors propose ATSSA, which replaces the "attend-to-everything" approach with a selective process. The core components include:
- Adaptive Threshold Generation: Instead of using a fixed sparsity pattern, the model uses a "matching matrix" to compare a specific character against the entire sequence. It then generates a unique threshold ($T_i$) for each query character.
- Differentiable Binarization: To make the selection process trainable, the authors use an approximate step function and Bernoulli sampling (via the Gumbel-Softmax trick). This allows the model to "decide" which keys to keep and which to discard while still being able to pass gradients during backpropagation.
- Top-k Constraint: To ensure the model doesn't accidentally discard too much information during early training, they implement a safety net that forces the model to keep at least the top $k$ most relevant keys (where $k=3$ was found to be optimal).
- Integration with FLAT: The mechanism is built on top of the FLAT (Flat-Lattice Transformer) architecture, which incorporates word-level lexicon information into character-level embeddings.
Key Insights
- Sparsity for Denoising: Unlike many sparse attention papers that aim for computational speedups, ATSSA uses sparsity as a tool for denoising. By ignoring irrelevant characters, the model produces a "sharper" attention map that better respects entity boundaries.
- Dynamic Focus: Previous attempts at sparse attention used fixed patterns (like attending only to neighbors). ATSSA is data-driven; the focus of the attention changes based on the specific meaning of the sentence.
- Faster Convergence: Interestingly, while the selective mechanism adds a tiny bit of overhead, the model actually converges faster than the baseline FLAT model because it isn't distracted by irrelevant character interactions.
Experimental Results
The model was evaluated on four standard benchmark datasets: Weibo NER, Resume NER, OntoNotes 4.0, and MSRA.
- Performance Boost: ATSSA achieved an average improvement of 1.68 F1 points over the baseline FLAT model across all datasets.
- State-of-the-Art: The model reached SOTA results on all four benchmarks, notably hitting 72.53 F1 on Weibo and 96.73 F1 on Resume.
- Efficiency: Despite the extra thresholding math, the model is only about 5% slower in inference speed than the standard FLAT model, making it highly practical for real-world deployment.
- Qualitative Success: Case studies showed that ATSSA correctly separated adjacent entities (like "Tencent" and "Lenovo") that standard Transformers often merged into a single, incorrect entity.
Conclusion
ATSSA demonstrates that selective attention is a powerful tool for improving the precision of Chinese NER. By dynamically filtering out noise, the model achieves superior boundary detection with minimal impact on inference speed. Future work may involve adapting this adaptive thresholding mechanism to other NLP tasks like text classification and natural language inference.
Summary updated