Local Additivity Based Data Augmentation for Semi-supervised NER

Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, Diyi Yang

Abstract

Named Entity Recognition (NER) is a fundamental stage in deep language understanding, yet current models rely heavily on extensive human-annotated data. To alleviate this dependency, we propose Local Additivity based Data Augmentation (LADA) for semi-supervised NER. LADA creates virtual training samples by interpolating between sequences that are semantically close to each other in the feature space. We introduce two primary variants: Intra-LADA, which interpolates tokens within a single sentence, and Inter-LADA, which samples different but semantically similar sentences for interpolation. We further extend LADA to the semi-supervised setting via a novel consistency loss for unlabeled data based on entity counts. Experiments on CoNLL 2003 and GermEval 2014 benchmarks demonstrate that LADA significantly outperforms strong baselines, particularly in low-resource scenarios.

Introduction: The Challenge of Sequence Labeling

While data augmentation techniques like "mixup"—which creates virtual training samples by linearly interpolating between random inputs and their labels—have been highly successful in image and text classification, they often struggle in sequence labeling tasks like NER. The primary obstacle is the sensitivity of NER to local context. Randomly mixing unrelated sentences often results in incoherent "Frankenstein" contexts that introduce excessive noise, confusing the model's ability to learn precise entity boundaries. LADA addresses this by making mixup "local," ensuring that interpolations occur only between semantically similar samples.

Methods: Local Additivity (LADA)

The core principle of LADA is ensuring that the two samples being mixed are "close" in the feature space. This locality constraint is vital for maintaining coherent context in sequence labeling.

Intra-LADA and Inter-LADA

  1. Intra-LADA: This variant performs interpolations among tokens within the same sentence. By shuffling word orders and interpolating hidden representations of tokens from the same word set, the model becomes more robust to structural variations without introducing external noise.
  2. Inter-LADA: Instead of random sampling, Inter-LADA selects a sentence and its $k$-nearest neighbors (kNN) using Sentence-BERT embeddings. This ensures the "mixed" context remains semantically coherent. The actual mixing occurs in the hidden layers of the encoder using a ratio $\lambda$ drawn from a Beta distribution.
Architecture of Inter-LADA showing interpolation between two semantically similar sentences.

Semi-supervised Learning with Semi-LADA

To leverage unlabeled data, we use back-translation to create paraphrases. Since paraphrases may have different lengths, preventing direct token-to-token alignment, we introduce a novel consistency loss: the model is trained to predict the same total number of entities for each category (e.g., 2 Persons, 1 Location) in both the original and paraphrased sentences. This entity count serves as a robust proxy for consistency when word alignments shift.

Experimental Results

We evaluated LADA on the CoNLL 2003 (English) and GermEval 2014 (German) benchmarks. To simulate low-resource environments, we tested models using 5%, 10%, and 30% of labeled training data alongside 10,000 unlabeled sentences.

Comparison of LADA variants against state-of-the-art baselines on CoNLL and GermEval datasets across different data sizes.

As shown in the results, LADA variants consistently outperform strong baselines. Notably, on CoNLL 2003 with only 5% labeled data, BERT + Semi-Intra-LADA achieved an F1 score of 87.15, a significant improvement over the base BERT and other semi-supervised techniques like Mean Teacher (MT + Noise). Visualization of loss curves also indicates that LADA acts as a powerful regularizer, preventing overfitting to small training sets.

Conclusion

LADA demonstrates that a locality constraint is vital for applying mixup to sequence labeling. By restricting interpolations to semantically similar neighborhoods, LADA provides a stable training signal that improves generalization in data-scarce settings. Future work will explore more sophisticated consistency losses that capture structural information beyond simple entity counts.

Summary updated

Abstract

Named Entity Recognition (NER) is one of the first stages in deep language understanding yet current NER models heavily rely on human-annotated data. In this work, to alleviate the dependence on labeled data, we propose a Local Additivity based Data Augmentation (LADA) method for semi-supervised NER, in which we create virtual samples by interpolating sequences close to each other. Our approach has two variations: Intra-LADA and Inter-LADA, where Intra-LADA performs interpolations among tokens within one sentence, and Inter-LADA samples different sentences to interpolate. Through linear additions between sampled training data, LADA creates an infinite amount of labeled data and improves both entity and context learning. We further extend LADA to the semi-supervised setting by designing a novel consistency loss for unlabeled data. Experiments conducted on two NER benchmarks demonstrate the effectiveness of our methods over several strong baselines. We have publicly released our code at https://github.com/GT-SALT/LADA.
BibTeX
@inproceedings{chen2020local,
  author = {Jiaao Chen and Zhenghui Wang and Ran Tian and Zichao Yang and Diyi Yang},
  title = {{Local Additivity Based Data Augmentation for Semi-supervised NER}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=TEfaFTZO1z},
}
Figure