Abstract
Named Entity Recognition (NER) is a fundamental stage in deep language understanding, yet current models rely heavily on extensive human-annotated data. To alleviate this dependency, we propose Local Additivity based Data Augmentation (LADA) for semi-supervised NER. LADA creates virtual training samples by interpolating between sequences that are semantically close to each other in the feature space. We introduce two primary variants: Intra-LADA, which interpolates tokens within a single sentence, and Inter-LADA, which samples different but semantically similar sentences for interpolation. We further extend LADA to the semi-supervised setting via a novel consistency loss for unlabeled data based on entity counts. Experiments on CoNLL 2003 and GermEval 2014 benchmarks demonstrate that LADA significantly outperforms strong baselines, particularly in low-resource scenarios.
Introduction: The Challenge of Sequence Labeling
While data augmentation techniques like "mixup"—which creates virtual training samples by linearly interpolating between random inputs and their labels—have been highly successful in image and text classification, they often struggle in sequence labeling tasks like NER. The primary obstacle is the sensitivity of NER to local context. Randomly mixing unrelated sentences often results in incoherent "Frankenstein" contexts that introduce excessive noise, confusing the model's ability to learn precise entity boundaries. LADA addresses this by making mixup "local," ensuring that interpolations occur only between semantically similar samples.
Methods: Local Additivity (LADA)
The core principle of LADA is ensuring that the two samples being mixed are "close" in the feature space. This locality constraint is vital for maintaining coherent context in sequence labeling.
Intra-LADA and Inter-LADA
- Intra-LADA: This variant performs interpolations among tokens within the same sentence. By shuffling word orders and interpolating hidden representations of tokens from the same word set, the model becomes more robust to structural variations without introducing external noise.
- Inter-LADA: Instead of random sampling, Inter-LADA selects a sentence and its $k$-nearest neighbors (kNN) using Sentence-BERT embeddings. This ensures the "mixed" context remains semantically coherent. The actual mixing occurs in the hidden layers of the encoder using a ratio $\lambda$ drawn from a Beta distribution.
Semi-supervised Learning with Semi-LADA
To leverage unlabeled data, we use back-translation to create paraphrases. Since paraphrases may have different lengths, preventing direct token-to-token alignment, we introduce a novel consistency loss: the model is trained to predict the same total number of entities for each category (e.g., 2 Persons, 1 Location) in both the original and paraphrased sentences. This entity count serves as a robust proxy for consistency when word alignments shift.
Experimental Results
We evaluated LADA on the CoNLL 2003 (English) and GermEval 2014 (German) benchmarks. To simulate low-resource environments, we tested models using 5%, 10%, and 30% of labeled training data alongside 10,000 unlabeled sentences.
As shown in the results, LADA variants consistently outperform strong baselines. Notably, on CoNLL 2003 with only 5% labeled data, BERT + Semi-Intra-LADA achieved an F1 score of 87.15, a significant improvement over the base BERT and other semi-supervised techniques like Mean Teacher (MT + Noise). Visualization of loss curves also indicates that LADA acts as a powerful regularizer, preventing overfitting to small training sets.
Conclusion
LADA demonstrates that a locality constraint is vital for applying mixup to sequence labeling. By restricting interpolations to semantically similar neighborhoods, LADA provides a stable training signal that improves generalization in data-scarce settings. Future work will explore more sophisticated consistency losses that capture structural information beyond simple entity counts.
Summary updated