Named Entity Recognition (NER)—the task of identifying people, places, and organizations in text—is a cornerstone of NLP. However, deep learning models for NER are notoriously "data-hungry," requiring massive amounts of human-annotated text to perform well. While data augmentation techniques like "Mixup" (linearly interpolating between data points) have revolutionized computer vision and text classification, they traditionally struggle with sequence labeling tasks like NER.
A paper from researchers at Georgia Tech, Google, and Citadel Securities, titled "Local Additivity Based Data Augmentation for Semi-supervised NER," introduces a method called LADA to bridge this gap. By constraining augmentations to local neighborhoods and introducing a novel consistency loss for unlabeled data, they significantly boost NER performance in low-data and semi-supervised settings.
The Challenge of Sequence Labeling
Data augmentation for NER is difficult because token-level labels are sensitive to context and order. In simple text classification, the model only needs to learn one label per sentence. In NER, the model must learn fine-grained relationships between every single token and its context.
Standard Mixup creates new samples by averaging the vectors of two random sentences, which often introduces too much noise. As visualized above, sequence labeling tasks have a more complex manifold where random interpolation often lands in "no man's land," creating gibberish that confuses the model rather than helping it generalize.
Methods: Local Additivity based Data Augmentation (LADA)
The authors propose focusing on "Local Additivity"—mixing things that are already semantically similar. They introduce two main variations:
- Intra-LADA: This interpolates between different tokens within the same sentence. By mixing a token's representation with its neighbors, the model becomes more robust to slight changes in word order and local context.
- Inter-LADA: This interpolates between different sentences. To avoid the noise problem, the authors use a $k$-Nearest Neighbors (kNN) approach (using Sentence-BERT) to find sentences that are semantically close to the target before interpolating them.
Semi-Supervised Learning with Entity Counting
To leverage unlabeled data, the authors use back-translation to create paraphrases. Since token alignment often breaks during translation, they designed a novel consistency loss. Instead of matching labels token-by-token, the model is penalized if the total count of each entity type (e.g., the number of "Person" entities) differs between the original sentence and its paraphrase. This global approach allows the model to learn from unlabeled data even when sentence structure is significantly altered.
Key Results and Insights
The researchers evaluated LADA on the CoNLL 2003 and GermEval 2014 benchmarks across various low-resource scenarios. LADA consistently outperformed strong baselines like vanilla BERT:
- Low-Data Boost: On CoNLL 2003 with only 5% labeled data, adding LADA to BERT improved the F1 score from 83.28 to 87.15.
- Effective Regularization: Even with 100% of the labeled data, LADA provided performance gains, acting as an effective regularizer that prevents overfitting.
- Robustness: The "Intra+Inter" combination proved to be the most stable strategy, suggesting that augmenting both local token context and broader sentence semantics is essential.
While powerful, LADA does require pre-calculating embeddings for the training corpus to find nearest neighbors, which adds computational overhead. However, its ability to significantly improve performance in data-scarce environments makes it a valuable tool for modern NER pipelines.
Citation: Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, Diyi Yang. "Local Additivity Based Data Augmentation for Semi-supervised NER." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.
Summary updated