Abstract
This paper evaluates the impact of Data Augmentation (DA) and Contrastive Learning (CL) on Named Entity Recognition (NER) performance across various dataset scales. Contrary to common assumptions, standard DA techniques often fail to improve upon strong transformer baselines. Supervised CL provides consistent gains but requires larger datasets to be effective. For low-resource scenarios, Contrastive Transfer Learning (CTL) is identified as the superior strategy.
Introduction
Named Entity Recognition (NER) is a fundamental NLP task that often suffers from data scarcity in specialized domains. While Data Augmentation (DA) and Contrastive Learning (CL) are popular remedies, their effectiveness on token-level tasks like NER—where label alignment is sensitive to structural changes—remains under-explored outside of extreme few-shot settings. This study benchmarks these methods using a RoBERTa baseline across small, medium, and large datasets.
Methodology
The researchers evaluated three primary approaches:
- Data Augmentation (DA): Techniques included round-trip translation (via NLLB), paraphrasing (via T5), and random token masking with a consistency loss (KL divergence).
- Supervised Contrastive Learning (CL): This approach treats contextualized token embeddings as training examples, maximizing similarity for tokens of the same entity type and minimizing it for different types.
- Contrastive Transfer Learning (CTL): A hybrid method where a model undergoes contrastive learning on a large source dataset before being fine-tuned on a smaller target dataset.
Results and Analysis
The experimental results across CoNLL-2003, OntoNotes v5, and Few-NERD datasets yielded several key insights:
The Limitations of Data Augmentation
DA methods generally failed to provide statistically significant improvements over the RoBERTa baseline. The authors suggest that because large-scale pre-trained models have already seen vast linguistic variations, simple augmentations like paraphrasing do not introduce novel, generalizable information.
Data Requirements for Contrastive Learning
While DA is often viewed as a low-resource tool, CL was found to be "data-hungry," requiring at least 5,000 sentences to show consistent benefits. On the full Few-NERD dataset, CL improved the F1 score from 67.90% to 68.65%.
The Superiority of CTL in Low-Resource Settings
For very small datasets (e.g., 100 sentences), CTL proved most effective. By leveraging a large source dataset for contrastive pre-training, the model develops more robust representations that transfer well to the target task.
Embedding Space Visualization
Analysis of token embeddings via t-SNE revealed that the "Non-Entity" (O) class significantly dominates the contrastive objective. While the model successfully separates named entities from non-entities, it struggles to distinguish between specific entity types (e.g., Person vs. Location). The model tends to optimize the loss by simply pushing the massive non-entity cluster away from all other tokens.
Conclusion
This research clarifies the roles of DA and CL in modern NER pipelines. DA is often redundant for strong transformer models, while CL is a valuable tool for larger datasets. For practitioners working with limited data, Contrastive Transfer Learning offers the most significant performance boost. Future work will focus on addressing class imbalance in CL to improve the separation of distinct entity categories.
Summary updated