Boundary Smoothing for Named Entity Recognition

Enwei Zhu, Jinpeng LiACL 2022 · DOI

Introduction

Named Entity Recognition (NER) is a fundamental task in natural language processing, where the goal is to identify and categorize entity spans within text. Modern state-of-the-art systems often employ "span-based" models that evaluate every possible start-and-end combination in a sentence. However, these models frequently suffer from the "over-confidence" problem. Because training typically uses "hard" targets—assigning a probability of 1.0 to the exact gold span and 0.0 to all others—models become excessively certain of their predictions, leading to poor calibration and a jagged loss landscape that hinders generalization.

Methodology: Boundary Smoothing

To address these issues, this research introduces Boundary Smoothing, a regularization technique inspired by label smoothing. Instead of a binary hard target, boundary smoothing redistributes a portion of the probability mass (denoted as $\epsilon$) from the gold entity span to its immediate spatial neighbors in the span grid.

Comparison between (a) traditional hard boundaries and (b) the proposed smoothed boundaries, where probability is shared with neighboring spans.

As illustrated in the figure above, the method uses Manhattan distance to define "surrounding spans." If a gold span is located at coordinates $(i, j)$, its probability is shared with spans whose start or end points are only a few tokens away. This approach acknowledges the inherent ambiguity in entity boundaries (e.g., whether to include a title like "The" or a suffix like "Jr.") and encourages the model to learn a smoother transition between entity and non-entity regions.

Experimental Results

The effectiveness of boundary smoothing was evaluated across eight benchmarks, including English (CoNLL 2003, OntoNotes 5, ACE 2004/2005) and Chinese (OntoNotes 4, MSRA, Weibo, Resume) datasets. The model, built on a RoBERTa/BERT backbone with a biaffine decoder, achieved new state-of-the-art results on most benchmarks, including an $F_1$ score of 93.65 on CoNLL 2003 and 87.15 on ACE 2005.

Key findings include:

  • Precision Gains: The technique primarily boosts precision by making the model more conservative and critical of its boundary predictions.
  • Spatial vs. Label Smoothing: Unlike traditional label smoothing (which spreads probability across classes), boundary smoothing spreads it across locations. The authors found that while label smoothing can actually hurt NER performance, boundary smoothing consistently improves it.

Model Calibration and Generalization

A significant advantage of boundary smoothing is the improvement in model calibration. Reliability diagrams demonstrate that as the smoothing parameter $\epsilon$ increases, the model's confidence aligns more closely with its actual precision, reducing the Expected Calibration Error (ECE).

Reliability diagrams for CoNLL 2003 and OntoNotes 5 showing how boundary smoothing ($ε > 0$) improves model calibration compared to the baseline ($ε = 0$).

Furthermore, visualizations of the loss landscape reveal that boundary smoothing leads to "flatter" minima. In deep learning theory, flatter minima are associated with better generalization, as the model becomes less sensitive to small perturbations in input data or parameters.

Conclusion and Limitations

Boundary smoothing provides a simple yet effective way to regularize span-based NER models, leading to better performance and more trustworthy confidence scores. However, the technique may slightly reduce recall in some scenarios, as the model becomes more critical of potential entities. Additionally, the optimal smoothing hyperparameters ($D$ and $\epsilon$) require tuning for specific datasets. Future work may extend this spatial smoothing logic to other structured prediction tasks like relation extraction.

Summary updated

BibTeX
@inproceedings{zhu2022boundary,
  author = {Enwei Zhu and Jinpeng Li},
  title = {{Boundary Smoothing for Named Entity Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.490},
}
Figure