Natural language is highly structured. When we manipulate text—whether to generate adversarial examples, augment training datasets, or obfuscate private information—we often swap individual words for synonyms. However, if we treat a sentence merely as a "bag of words" and replace tokens based purely on semantic similarity, the underlying grammar frequently collapses.
To prevent this, a powerful and intuitive mechanism has emerged across various NLP domains: part-of-speech (POS) constrained substitution. The core intuition is that every sentence possesses a syntactic skeleton. By ensuring that a replacement token shares the exact same grammatical category (or POS tag) as the original word, we can alter the lexical content of a sentence while preserving its structural integrity.
Why Naive Substitution Fails
Standard word embeddings (such as Word2Vec or BERT) map words into a continuous vector space based on contextual co-occurrence. While these spaces excel at capturing semantic proximity, they often conflate different syntactic roles. For instance, the noun flight and the verb fly are close neighbors in an embedding space because they appear in similar contexts. However, swapping a noun for a verb instantly breaks a sentence's grammar.
Unconstrained lexical substitution introduces structural disruptions that create two major issues:
- Human Detectability: The text sounds unnatural or ungrammatical to human readers.
- Machine Disruption: Downstream models (like parsers or classifiers) fail to process the sentence correctly because the expected grammatical patterns are broken.
The Mechanism: Preserving the Structural Skeleton
POS-constrained substitution acts as a hard filter on top of semantic search spaces. Instead of searching the entire vocabulary or an open geometric neighborhood for a replacement token, the substitution algorithm executes a two-step process:
- Tagging: It identifies the exact POS tag of the target word within its specific context (e.g., distinguishing between run as a noun versus run as a verb).
- Filtering: It restricts the candidate pool exclusively to words that match that exact syntactic slot.
Original: The quick brown fox jumps over the lazy dog.
Tags: DET ADJ ADJ NOUN VERB ADP DET ADJ NOUN
│ │
Constraint: ▼ ▼
Allowed: [wolf] [leaps]
Blocked: [run] [jumping]
This approach works because it decouples syntax from semantics. The sentence's structural frame remains valid, allowing downstream language models or human readers to process the syntactic relationships correctly, even if the specific meaning shifts or becomes obfuscated.
Crafting Imperceptible Adversarial Attacks
In adversarial machine learning, the goal is to fool a model without a human noticing the modification. If an attack introduces ungrammatical phrasing, it is easily flagged.
To expose models that rely on simple word-counting heuristics rather than true syntactic understanding, Yuan Zhang et al. developed the PAWS dataset in PAWS: Paraphrase Adversaries from Word Scrambling. They used POS tagging and language model scoring to generate sentence pairs that share identical word sets but differ syntactically, forcing models to look beyond vocabulary overlap.
When executing black-box attacks on text classifiers, maintaining fluency is paramount. In Generating Natural Language Attacks in a Hard Label Black Box Setting, Rishabh Maheshwary et al. applied POS filtering to restrict the search space to linguistically valid synonyms, preventing the generation of nonsensical text. Similarly, James Wang et al. introduced the RAFT framework in RAFT: Realistic Attacks to Fool Text Detectors, demonstrating that enforcing POS consistency allows machine-generated text to bypass automated detectors while remaining completely natural to human readers.
For security vulnerabilities like backdoor injections, Fanchao Qi et al. leveraged shared sememes and POS tags from structured knowledge bases in Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution to inject triggers that remain entirely invisible to automated defenses.
Beyond attacks, this mechanism allows for the efficient generation of text counterfactuals—small edits that flip a model's prediction for interpretability purposes. Dimitris Lymperopoulos et al. showed in Optimal and efficient text counterfactuals using Graph Neural Networks that enforcing POS constraints during word substitution allows graph neural networks to find optimal counterfactual edits without the massive computational overhead of large generative language models.
High-Fidelity Data Augmentation and Translation
In data-scarce environments, generating synthetic training data requires introducing lexical diversity without invalidating the ground-truth labels.
Early phrase-based statistical machine translation systems faced severe bottlenecks when encountering unseen phrases. To resolve this, Zhongjun He et al. introduced a template-matching strategy in Partial Matching Strategy for Phrase-based Statistical Machine Translation that enforced strict POS sequence matching, ensuring that when an unseen phrase was mapped to a template, the syntactic skeleton remained valid.
For modern neural models, Hyeonseok Moon et al. demonstrated in Empirical Analysis of Noising Scheme based Synthetic Data Generation for Automatic Post-editing that injecting noise by swapping words exclusively with matching POS tags creates realistic, "human-mimicking" errors that significantly improve automatic post-editing models.
This strategy is particularly vital in complex linguistic settings like code-switching (mixing multiple languages within a single sentence). Sangmin Lee et al. developed the UniCoM framework in UniCoM: A Universal Code-Switching Speech Generator, showing that substituting words based on their POS tags preserves naturalness and semantic meaning more effectively than phrase-level substitutions.
Similarly, Kartik demonstrated in Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation that selectively substituting specific categories like nouns and adjectives—while leaving complex grammatical structures like verbs intact—prevents morphological inconsistencies and structural disruptions across languages.
Utility-Preserving Privacy and Watermarking
When text must be modified for security or tracking, POS constraints ensure the output remains useful for downstream tasks.
In user-side privacy management, Ryoma Sato proposed the PRISM framework in Making Translators Privacy-aware on the User's Side. PRISM replaces sensitive words with placeholders that match the original POS and grammatical context. This allows external translation services to correctly interpret the sentence structure, ensuring an accurate translation that can later be locally de-anonymized.
At the model level, Yansong Li et al. found in Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Services that restricting privacy perturbations to specific POS categories (like nouns and verbs) preserves text semantics and syntax far better than indiscriminate embedding noise.
Original text: The patient was diagnosed with diabetes at Mayo Clinic.
Indiscriminate: The patient was running with blue at quickly table. (Unparseable)
POS-Constrained: The individual was diagnosed with illness at local facility. (Parseable)
For text watermarking, György Szarvas et al. demonstrated in Uncertainty Detection for Natural Language Watermarking that restricting synonym substitution to a small, curated set of high-frequency words (such as uncertainty cues) minimizes contextual errors. This constraint keeps the watermark completely imperceptible while maintaining high grammatical precision.
Key Foundational Work
A Rote Extractor with Edit Distance-Based Generalisation and Multi-Corpora Precision Calculation (2006) — contributed 2 concept elements to this direction.
Partial Matching Strategy for Phrase-based Statistical Machine Translation (2008) — contributed 1 concept elements to this direction.
Combining Multiple Resources to Improve SMT-based Paraphrasing Model (2008) — contributed 1 concept elements to this direction.