Abstract
Span Identification (SpanID) tasks, such as Named Entity Recognition (NER) and Aspect-Based Sentiment Analysis (ABSA), often suffer from model over-fitting to superficial label-to-span mappings and data scarcity for long-tailed categories. We present PeerDA, a novel data augmentation strategy that models the "Peer" (PR) relation—the relationship between two spans of the same category. By reframing SpanID as a Machine Reading Comprehension (MRC) task where queries are peer spans rather than category labels, PeerDA forces models to learn deep semantic representations. Our approach achieves state-of-the-art results on six out of ten benchmarks and demonstrates significant gains in low-resource and cross-domain settings.
Introduction
Span Identification (SpanID) is a fundamental component of natural language processing. Traditionally, these models are trained on the Subordinate (SUB) relation, which maps a specific span to a category (e.g., identifying "London" as a "Location"). However, this paradigm faces two major bottlenecks:
- Over-fitting: Models often memorize superficial mappings instead of understanding context, leading to poor generalization in cross-domain scenarios.
- Data Scarcity: Rare or "long-tailed" categories lack sufficient examples for robust training.
Methodology: The Peer Relation Paradigm
The core innovation of PeerDA is the shift from SUB relations to Peer (PR) relations. If Span A and Span B both belong to the same category, they are considered "Peers."
1. MRC Paradigm Shift
PeerDA treats SpanID as a Machine Reading Comprehension (MRC) problem. While a standard MRC query might ask the model to "Highlight the parts related to [Category]," a PeerDA query asks it to "Highlight the parts similar to [Peer Span]." By using a span (e.g., "Hawaii") as the query instead of a label (e.g., "LOC"), the model is forced to leverage span semantics, preventing it from relying on fixed category-label memorization.
2. Quadratic Data Augmentation
While SUB relations grow linearly with the number of labels, PR relations grow quadratically. This provides a massive pool for data augmentation, allowing for the creation of "category-balanced" training sets by over-sampling peer pairs for rare categories.
3. Model Enhancements
To ensure efficiency, the authors introduced Bilinear Scoring, which reduces the memory footprint by over 4x compared to standard concatenation methods. Additionally, a Max-Min Contrastive Loss strategy was implemented to explicitly separate the representations of confusing positive and negative spans in the semantic space.
Experimental Results
The authors evaluated PeerDA on ten datasets across four tasks (NER, ABSA, Contract Clause Extraction, and Propaganda Detection) and seven domains.
Performance and Efficiency
PeerDA achieved state-of-the-art (SOTA) results on 6 out of 10 datasets. Notably, in the legal domain (CUAD dataset), PeerDA outperformed the previous best model by 8.7 AUPR.
As shown in the learning curves above, PeerDA demonstrates exceptional data efficiency. In many cases, PeerDA trained on only 50% of the data matched or exceeded the performance of a standard MRC model trained on 100% of the data.
Cross-Domain Robustness
In out-of-domain evaluations (e.g., training on news and testing on social media), PeerDA significantly outperformed standard MRC models. This confirms that the peer-relation training forces the model to learn generalized semantic features rather than domain-specific label mappings.
Conclusion and Future Work
PeerDA introduces a powerful regularization effect by prioritizing semantics over memorization. While highly effective, the current implementation is focused on encoder-only models and relies on existing labeled spans. Future research will explore extending this framework to encoder-decoder architectures (like T5) and integrating external knowledge bases to further disambiguate similar categories.
Summary updated