Abstract
Named Entity Recognition (NER) is a critical component of NLP pipelines but requires expensive, dense token-level annotations. In low-resource scenarios, models frequently overfit due to data scarcity. This paper presents RoPDA (Robust Prompt-based Data Augmentation), a novel framework that utilizes pre-trained language models with continuous "soft" prompts to generate diverse, high-quality synthetic training data. RoPDA introduces five specialized augmentation operations, including strategic "label-flipping" to create adversarial examples, and employs a self-consistency filter to ensure data quality. Experimental results across multiple benchmarks demonstrate that RoPDA consistently outperforms existing data augmentation and semi-supervised methods in few-shot settings.
Introduction
The primary challenge in Named Entity Recognition (NER) is the high cost of dense, token-level annotations. Producing these is both time-consuming and expensive, leading to significant performance degradation in low-resource domains where only a handful of examples per entity type may be available. While data augmentation (DA) is a common solution, traditional methods like simple word-swapping often compromise syntactic integrity or fail to capture contextual nuances. RoPDA addresses these limitations by generating context-aware synthetic data using a frozen pre-trained model enhanced with trainable soft prompts.
The RoPDA Framework
The RoPDA framework operates through a multi-stage pipeline designed to maximize the utility of limited gold-standard data while maintaining high linguistic quality.
1. Linearization and Prompting
Standard NER tags are converted into a natural language format (e.g., [ David | person ]). RoPDA utilizes a T5 (Seq2Seq) backbone with "Soft Prompts"—trainable continuous vectors added to each layer. By tuning only these vectors while keeping the massive T5 model frozen, the framework avoids the catastrophic forgetting typically associated with full fine-tuning on small datasets.
2. Augmentation Operations
The framework performs five distinct operations to diversify the training set:
- Label-Preserving (Op1): Regenerates entities of the same type or augments the surrounding context.
- Label-Flipping (Op2): Strategically changes an entity's type, creating adversarial examples that challenge the model to learn more precise decision boundaries.
- Adding (Op3): Inserts new entities into the context.
- Erasing (Op4): Removes entities while maintaining a coherent sentence structure.
- Context (Op5): Modifies the surrounding text while keeping the entity and its label fixed.
3. Self-Consistency Filtering
To maintain high data quality and filter out noisy generations, RoPDA implements a bidirectional verification process.
This filter checks if the model can consistently predict the word given the label (Type2Word) and the label given the word (Word2Type). Samples that fail this "sanity check" are discarded, ensuring that the synthetic data remains reliable for training.
4. Adversarial Mixup
Directly training on label-flipped data can sometimes introduce instability. RoPDA mitigates this by using Mixup, a technique that linearly interpolates the hidden representations of synthetic adversarial samples with original samples. This results in a smoother label distribution and enhanced model robustness.
Key Insights
- Label-Flipping as Adversarial Training: Changing a label while preserving context (or vice versa) acts as an adversarial attack. It teaches the model exactly which contextual clues distinguish a "Person" from an "Organization."
- Context is King: Augmenting the words surrounding an entity is as vital as augmenting the entities themselves. Without context diversity, the model simply memorizes entity lists rather than learning generalizable patterns.
- Automated Sanity Checks: The bidirectional self-consistency filter acts as an automated quality control layer, significantly reducing the manual labor usually required to clean synthetic datasets.
Experimental Results
RoPDA was evaluated on three diverse benchmarks: CoNLL03 (news), MIT Restaurant, and MIT Movie. Testing in "shot-K" settings (5, 10, 20, and 50 samples per entity type), RoPDA consistently outperformed strong baselines like SDANER and PromDA.
On the CoNLL03 dataset, RoPDA improved F1-scores by 2.3% to 8.3% over the baseline. Furthermore, when unlabeled data was included (RoPDA*), it outperformed MetaST, a leading semi-supervised method. This suggests that high-quality synthetic data can be more effective than standard semi-supervised learning on raw, unlabeled text.
Conclusion
RoPDA provides a robust solution for NER in data-scarce environments. By combining soft-prompted generation with adversarial label-flipping and rigorous self-consistency filtering, it generates high-quality synthetic data that significantly boosts model performance. Future work will focus on refining label-flipping strategies and reducing the computational overhead of the generation-filtering pipeline.
Summary updated