PeerDA: Data Augmentation via Modeling Peer Relation for Span Identification Tasks

Weiwen Xu, Xin Li, Yang Deng, Wai Lam, Lidong Bing

Abstract

Span Identification (SpanID) tasks, such as Named Entity Recognition (NER) and Aspect-Based Sentiment Analysis (ABSA), often suffer from model over-fitting to superficial label-to-span mappings and data scarcity for long-tailed categories. We present PeerDA, a novel data augmentation strategy that models the "Peer" (PR) relation—the relationship between two spans of the same category. By reframing SpanID as a Machine Reading Comprehension (MRC) task where queries are peer spans rather than category labels, PeerDA forces models to learn deep semantic representations. Our approach achieves state-of-the-art results on six out of ten benchmarks and demonstrates significant gains in low-resource and cross-domain settings.

Introduction

Span Identification (SpanID) is a fundamental component of natural language processing. Traditionally, these models are trained on the Subordinate (SUB) relation, which maps a specific span to a category (e.g., identifying "London" as a "Location"). However, this paradigm faces two major bottlenecks:

  1. Over-fitting: Models often memorize superficial mappings instead of understanding context, leading to poor generalization in cross-domain scenarios.
  2. Data Scarcity: Rare or "long-tailed" categories lack sufficient examples for robust training.

Methodology: The Peer Relation Paradigm

The core innovation of PeerDA is the shift from SUB relations to Peer (PR) relations. If Span A and Span B both belong to the same category, they are considered "Peers."

Figure 1: Illustration of Subordinate (SUB) and Peer (PR) relations within the SpanID hierarchy.

1. MRC Paradigm Shift

PeerDA treats SpanID as a Machine Reading Comprehension (MRC) problem. While a standard MRC query might ask the model to "Highlight the parts related to [Category]," a PeerDA query asks it to "Highlight the parts similar to [Peer Span]." By using a span (e.g., "Hawaii") as the query instead of a label (e.g., "LOC"), the model is forced to leverage span semantics, preventing it from relying on fixed category-label memorization.

2. Quadratic Data Augmentation

While SUB relations grow linearly with the number of labels, PR relations grow quadratically. This provides a massive pool for data augmentation, allowing for the creation of "category-balanced" training sets by over-sampling peer pairs for rare categories.

3. Model Enhancements

To ensure efficiency, the authors introduced Bilinear Scoring, which reduces the memory footprint by over 4x compared to standard concatenation methods. Additionally, a Max-Min Contrastive Loss strategy was implemented to explicitly separate the representations of confusing positive and negative spans in the semantic space.

Experimental Results

The authors evaluated PeerDA on ten datasets across four tasks (NER, ABSA, Contract Clause Extraction, and Propaganda Detection) and seven domains.

Performance and Efficiency

PeerDA achieved state-of-the-art (SOTA) results on 6 out of 10 datasets. Notably, in the legal domain (CUAD dataset), PeerDA outperformed the previous best model by 8.7 AUPR.

Figure 6: Performance comparison between standard MRC and PeerDA across different training data percentages (10% to 100%).

As shown in the learning curves above, PeerDA demonstrates exceptional data efficiency. In many cases, PeerDA trained on only 50% of the data matched or exceeded the performance of a standard MRC model trained on 100% of the data.

Cross-Domain Robustness

In out-of-domain evaluations (e.g., training on news and testing on social media), PeerDA significantly outperformed standard MRC models. This confirms that the peer-relation training forces the model to learn generalized semantic features rather than domain-specific label mappings.

Conclusion and Future Work

PeerDA introduces a powerful regularization effect by prioritizing semantics over memorization. While highly effective, the current implementation is focused on encoder-only models and relies on existing labeled spans. Future research will explore extending this framework to encoder-decoder architectures (like T5) and integrating external knowledge bases to further disambiguate similar categories.

Summary updated

Abstract

Span identification aims at identifying specific text spans from text input and classifying them into pre-defined categories. Different from previous works that merely leverage the Subordinate (SUB) relation (i.e. if a span is an instance of a certain category) to train models, this paper for the first time explores the Peer (PR) relation, which indicates that two spans are instances of the same category and share similar features. Specifically, a novel Peer Data Augmentation (PeerDA) approach is proposed which employs span pairs with the PR relation as the augmentation data for training. PeerDA has two unique advantages: (1) There are a large number of PR span pairs for augmenting the training data. (2) The augmented data can prevent the trained model from over-fitting the superficial span-category mapping by pushing the model to leverage the span semantics. Experimental results on ten datasets over four diverse tasks across seven domains demonstrate the effectiveness of PeerDA. Notably, PeerDA achieves state-of-the-art results on six of them.
BibTeX
@inproceedings{xu2023peerda,
  author = {Weiwen Xu and Xin Li and Yang Deng and Wai Lam and Lidong Bing},
  title = {{PeerDA: Data Augmentation via Modeling Peer Relation for Span Identification Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.484},
}
Figure