Introduction
Neural networks in Natural Language Processing (NLP) are notoriously brittle; small, intentional perturbations to input data—known as adversarial attacks—can easily degrade model performance. Adversarial Training (AT) has emerged as a primary defense, involving the addition of noise during training to improve robustness. In NLP, where inputs are discrete, researchers often employ Virtual Adversarial Training (VAT) to apply noise to continuous word embeddings. However, standard VAT methods suffer from two primary limitations: they reset noise randomly for every batch (the initialization problem) and apply a uniform noise budget across all tokens regardless of their semantic importance (the constraint problem).
Methodology: The TA-VAT Approach
This paper introduces Token-Aware Virtual Adversarial Training (TA-VAT), a method designed to craft more precise and stable adversarial noise by accounting for token-specific characteristics. TA-VAT introduces two core innovations:
- Global Perturbation Vocabulary ($V$): Instead of initializing noise randomly for each training step, TA-VAT maintains a global dictionary that stores and accumulates perturbations for every word in the vocabulary. This provides a "warm start" for noise generation, reducing random jitter and stabilizing the training process.
- Token-Level Constraints and Scaling: Rather than applying a single noise budget to an entire sequence, TA-VAT applies constraints at the individual token level. It utilizes a Scaling Index ($n^i$) based on the gradient magnitude of each token. This allows "important" tokens—those that significantly influence the model's output—to receive a larger perturbation budget, while trivial tokens are more tightly constrained.
Experimental Results
TA-VAT was evaluated across the GLUE benchmark, showing consistent improvements over baseline models like BERT and ALBERT. By focusing perturbations on the most critical parts of the sentence, the model learns a more robust representation of language.
As shown in the results above, TA-VAT significantly outperforms standard BERT and state-of-the-art adversarial methods like FreeLB. On the GLUE benchmark, TA-VAT boosted the BERT-base score from 78.3 to 80.9 on the test set and improved the ALBERT-xxlarge model to 90.9 on the development set.
Data Efficiency and Robustness
One of the most striking findings is TA-VAT's performance in low-resource settings. The method is particularly effective when training data is scarce, as the global perturbation vocabulary allows the model to leverage "memory" of token sensitivity across different contexts.
Beyond standard benchmarks, TA-VAT demonstrates enhanced robustness against specialized attacks. When subjected to the TextFooler attack, TA-VAT increased the "Attacked Accuracy" and nearly doubled the number of queries required for an attacker to successfully break the model. Furthermore, the learned perturbation vocabulary proved to be transferable; applying the learned noise to a vanilla model's embeddings improved its performance even without additional training.
Conclusion
TA-VAT addresses the inherent limitations of uniform adversarial training in NLP by introducing token-awareness. By maintaining a global memory of perturbations and scaling noise based on token importance, TA-VAT achieves superior performance on GLUE, better data efficiency, and increased robustness against adversarial attacks. While the method requires additional computational steps and memory to maintain the perturbation vocabulary, the gains in stability and accuracy make it a compelling approach for robust natural language understanding.
Summary updated