Substructure Substitution: Structured Data Augmentation for NLP

Haoyue Shi, Karen Livescu, Kevin Gimpel

Abstract

We study a family of data augmentation methods, substructure substitution (SUB2), for natural language processing (NLP) tasks. SUB2 generates new examples by substituting substructures (e.g., subtrees or subsequences) with ones with the same label, which can be applied to many structured NLP tasks such as part-of-speech tagging and parsing. For more general tasks (e.g., text classification) which do not have explicitly annotated substructures, we present variations of SUB2 based on constituency parse trees, introducing structure-aware data augmentation methods to general NLP tasks. For most cases, training with the augmented dataset by SUB2 achieves better performance than training with the original training set. Further experiments show that SUB2 has more consistent performance than other investigated augmentation methods, across different tasks and sizes of the seed dataset.

Introduction

Data augmentation is a critical technique for training robust machine learning models, especially when annotated data is scarce. In the field of Natural Language Processing (NLP), common techniques typically fall into two categories: token-level methods (e.g., synonym swapping) and global generation methods (e.g., back-translation). However, these approaches often overlook the underlying structural hierarchy of language. Token-level swaps can disrupt local syntax, while global generation can be computationally intensive and prone to hallucinations.

This paper introduces Substructure Substitution ($SUB^2$), a data augmentation family that seeks a middle ground by leveraging the structural components of language. $SUB^2$ generates diverse, syntactically plausible new examples by swapping sentence parts—such as subtrees or sequences—with others that share the same label, without requiring heavy generative models.

Methodology

The fundamental principle of $SUB^2$ is to identify a substructure within a sentence and replace it with a different substructure that possesses the same label.

Structured Tasks

In tasks where labels are already associated with sentence parts (e.g., POS tagging or dependency parsing), the labels are inherent. For instance, in POS tagging, a sequence of words tagged as DT NN (e.g., "a book") can be substituted with another DT NN sequence (e.g., "an orange").

SUB2 applied to POS tagging by swapping sequences with identical tag patterns (DT NN).

General Tasks

For tasks like text classification that lack explicit substructures, the authors utilize a constituency parser to generate trees for the training data. They then apply various constraints for swapping, such as $SUB^2+N$ (equal word length), $SUB^2+P$ (valid phrases), $SUB^2+L$ (identical syntactic labels), and $SUB^2+T$ (same classification label).

Experimental Evaluation

The authors evaluated $SUB^2$ across diverse benchmarks, including Universal Dependencies (UD) for POS tagging, the Penn Treebank (PTB) for dependency parsing, and the Stanford Sentiment Treebank (SST) for text classification.

Key Results

  • Low-Resource POS Tagging: Combined with pretrained XLM-R models, $SUB^2$ established a new state-of-the-art for low-resource languages in the UD dataset.
  • Few-Shot Parsing: In constituency parsing with only 50 training examples, $SUB^2$ improved the F1 score from 33.1 to 71.8 on English Foreebank—a massive jump compared to other augmentation methods.
  • Sentiment Analysis: On the SST dataset, $SUB^2$ helped achieve a new state-of-the-art test accuracy of 56.6%, outperforming previous specialized models.

Discussion and Insights

The "Colorless Green Ideas" Effect

A significant finding is that $SUB^2$ does not require the generated sentences to be semantically coherent. As illustrated in Figure 2, swapping objects can result in semantically unusual sentences like "I read milk" or "The cat love books." Surprisingly, these syntactically correct but semantically "weird" examples still help the model learn superior representations.

Examples of generated sentences in dependency parsing showing the 'Colorless Green Ideas' effect.

Consistency and Structure

Unlike contextualized word substitution, which can introduce excessive noise, $SUB^2$ demonstrated remarkable consistency across different tasks and dataset sizes. Furthermore, using linguistically motivated constituency parse trees as a backbone for augmentation significantly outperformed arbitrary "balanced trees," underscoring the importance of respecting syntax.

Conclusion

$SUB^2$ provides a robust, structure-aware framework for NLP data augmentation. While it relies on the availability of a reliable parser for general tasks, its ability to improve performance across structured and unstructured tasks—even in low-resource settings—makes it a valuable tool. Future work may explore the regularization effects of random shuffling in few-shot parsing and extend $SUB^2$ to tasks like natural language inference or question answering.

Summary updated

Abstract

We study a family of data augmentation methods, substructure substitution (SUB2), for natural language processing (NLP) tasks. SUB2 generates new examples by substituting substructures (e.g., subtrees or subsequences) with ones with the same label, which can be applied to many structured NLP tasks such as part-of-speech tagging and parsing. For more general tasks (e.g., text classification) which do not have explicitly annotated substructures, we present variations of SUB2 based on constituency parse trees, introducing structure-aware data augmentation methods to general NLP tasks. For most cases, training with the augmented dataset by SUB2 achieves better performance than training with the original training set. Further experiments show that SUB2 has more consistent performance than other investigated augmentation methods, across different tasks and sizes of the seed dataset.
BibTeX
@inproceedings{shi2021substructure,
  author = {Haoyue Shi and Karen Livescu and Kevin Gimpel},
  title = {{Substructure Substitution: Structured Data Augmentation for NLP}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=ZHEaCtMvuHv},
}
Figure