Semantic Preserving Data Augmentation for Discrete Text

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2123 papers133 conceptsfrozen 2026-10-04

In computer vision, data augmentation is almost "free." If you flip an image of a cat, it is still a cat. If you crop it or slightly adjust the brightness, the semantic label remains invariant. However, as Steven Y. Feng and colleagues note in A Survey of Data Augmentation Approaches for NLP, natural language is fundamentally different because it is discrete and fragile.

In the world of pixels, a small change is a small change. In the world of tokens, a small change—like adding the word "not" or swapping "plaintiff" for "defendant"—can completely invert the meaning of a sentence. This "semantic fragility" is the primary hurdle for researchers trying to expand training sets for NLP models.

The Failure of Token Level Perturbations

Early attempts at NLP augmentation relied on simple "text-editing" heuristics: swapping words with synonyms, deleting random tokens, or changing word order. While these methods can work for simple classification, they often fail in complex domains.

For instance, Md Rizwan Parvez and Wasi Uddin Ahmad found that in legal privacy policies, even minor phrasing changes can alter legal obligations, making standard synonym replacement dangerous Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policies. Similarly, Yichun Yin argues that these discrete swaps often result in ungrammatical "word salad" that confuses rather than helps the model Dialog State Tracking with Reinforced Data Augmentation.

Walking the Semantic Manifold

To solve this, researchers have moved away from editing raw text and toward "walking the manifold"—generating new examples by navigating the continuous vector space where words live.

One influential approach is SSMBA, developed by Nathan Ng and Kyunghyun Cho. Instead of random swaps, SSMBA uses a "corruption-reconstruction" pair. It slightly perturbs a sentence and then uses a Masked Language Model (MLM) to "reconstruct" it back onto the data manifold SSMBA: Self-Supervised Manifold Based Data Augmentation. This ensures the synthetic sentence remains realistic and semantically related to the original.

Another strategy is Mixup, adapted from vision to NLP by Yongyi Mao and Hongyu Guo. Rather than creating a new sentence, Mixup creates a "synthetic representation" by linearly interpolating the embeddings of two different sentences and their labels Augmenting Data with Mixup for Sentence Classification. This forces the model to learn smoother decision boundaries in the continuous space, bypassing the "jumpy" nature of discrete tokens.

Structural and Generative Invariance

If tokens are too fragile to touch, perhaps we can manipulate the structure of the language. Gözde Gül Şahin and Mark Steedman pioneered "Dependency Tree Morphing," which applies vision-inspired concepts like cropping and rotating to the syntactic backbone of a sentence Data Augmentation via Dependency Tree Morphing for Low-Resource Languages. By moving sub-trees around, they create new valid sentences that preserve the core grammatical logic.

Other researchers use high-level generative tasks as a form of augmentation. Tengfei Ma uses text summarization to create "contrastive samples." A summary is, by definition, a semantically consistent but lexically different version of the original text, making it an ideal "positive pair" for contrastive learning Constructing contrastive samples via summarization for text classification.

Augmentation via Internal Noise

Finally, some of the most effective modern techniques don't change the input at all. Instead, they perturb the model's internal state.

Hooman Sedghamiz and colleagues demonstrated that using a Transformer's internal dropout masks can act as a powerful data augmentation tool SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence Representations. By passing the same sentence through the network twice with different dropout patterns, the model sees two slightly different "views" of the same data. This provides the necessary variance for contrastive learning without the risk of destroying the sentence's meaning through token deletion.

This shift—from "editing strings" to "perturbing representations"—represents the core evolution of the field. As Tianduo Wang and Wei Lu 0011 argue with their DiffAug framework, the future of NLP augmentation lies in making these transformations differentiable and learnable, rather than heuristic and destructive Differentiable Data Augmentation for Contrastive Sentence Representation Learning.

Papers in this version

Showing 60 of 123 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 142 papers
  2. Version 1 · 2026-10-04 · 123 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_discrete_data_augmentation_constraints_bdec94de,
  title = {discrete data augmentation constraints},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-data-augmentation-constraints/rdr_bdec94de82624cf39f70c4bd8aea6b63},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_preserving_data_augmentation_fo_bdec94de_v1,
  title = {Semantic Preserving Data Augmentation for Discrete Text},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-preserving-data-augmentation-for-discrete-text/rdr_bdec94de82624cf39f70c4bd8aea6b63/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}