Word Embedding Perturbation for Sentence Classification

Dongxu Zhang, Zhichao YangarXiv 2018 · OpenReview

TL;DR

This paper tackles the common problem of overfitting in natural language processing (NLP) models, especially when training data is scarce. The authors propose perturbing word embeddings with various types of noise (Gaussian, Bernoulli, adversarial) and applying constraints to these perturbations. Their experiments show that these data augmentation techniques can consistently improve sentence classification performance.

What Problem Does This Paper Address?

Training robust NLP models often requires large amounts of labeled data. However, acquiring such data is expensive and time-consuming, leading to insufficient training sets. When models are trained on limited data, they tend to "overfit," meaning they learn the training data too well, including its noise and specific quirks, and consequently perform poorly on unseen data. Data augmentation, a technique widely used in computer vision to generate more training examples, is less straightforward in NLP because language is discrete. Previous NLP augmentation methods often relied on external knowledge bases or complex NLP tools. This paper aims to bridge this gap by exploring simpler, yet effective, data augmentation strategies directly on the continuous word embedding space, which is a crucial component of modern NLP models.

Methods

The core idea is to introduce "noise" into the word embedding layer of a neural network during training. This is akin to slightly altering the input representations to make the model more robust. The paper explores several types of noise:

  • Gaussian Noise: Directly adding random noise sampled from a Gaussian distribution to the word embeddings.
  • Bernoulli Noise (Dropout): Randomly setting some word embedding units to zero with a certain probability, inspired by the successful dropout technique in deep learning.
  • Adversarial Training: Perturbing the word embeddings in a direction that maximally increases the model's loss function. This forces the model to learn representations that are less sensitive to small, adversarial changes.

Beyond these basic noise types, the authors also introduce variants with constraints:

  • Word Dropout: Randomly replacing entire word embeddings with a special "unknown" token embedding.
  • Semantic Dropout: Applying dropout to individual dimensions of the word embedding, assuming different dimensions capture different semantic meanings.
  • Gaussian/Bernoulli Adversarial Noise: Combining adversarial training with Gaussian or Bernoulli noise.

The key innovation lies in applying these perturbations directly to the continuous word embedding space, rather than manipulating discrete words or relying on external linguistic resources.

Datasets

The paper evaluates its methods on several standard NLP datasets for sentence classification tasks:

  • MR (Movie Review): Sentiment analysis.
  • SST2 (The Stanford Sentiment Treebank): Sentiment analysis.
  • CR (Customer Review): Sentiment analysis.
  • TREC (Question Type): Classifying the type of question.
  • RE (Relation Classification): Identifying semantic relations between entities.
  • TrecQA (Answer Selection): Determining if a question-answer pair is relevant.

The authors note that these datasets are often used in scenarios with limited training data, making them suitable for evaluating data augmentation techniques.

Key Insights and Ideas

  • Word Embeddings as a Target for Augmentation: Perturbing the continuous word embedding space is a viable and effective strategy for data augmentation in NLP, circumventing the discrete nature of words.
  • Adversarial Training is a Robust Choice: Vanilla adversarial training consistently improved performance across all tested datasets, highlighting its effectiveness in making models more robust.
  • Constraints Can Enhance Noise: Applying constraints, such as semantic dropout or adversarial constraints, can further improve the robustness and performance of noise-based augmentation methods.
  • Continuous Noise Outperforms Discrete: Continuous noise (like Gaussian) generally performed as well as or better than discrete noise (like Bernoulli dropout), potentially due to higher entropy which aids model learning.
  • Augmentation is More Crucial with Less Data: The benefits of these augmentation techniques become more pronounced as the amount of available training data decreases, directly addressing the overfitting problem.

Key Results

The paper reports consistent improvements across various sentence classification tasks. For instance, Adversarial Training showed improvements of 0.1% to 0.4% across all datasets. Gaussian adversarial noise led to significant gains on the SST and TrecQA datasets. While specific percentage improvements vary by dataset and method, the general trend is that these perturbations either improve performance or, at worst, do not significantly harm it. The results are presented in Table 2, showing accuracy for MR, SST2, CR, TREC, and F1 for RE, and MAP for TrecQA, along with an "Overall" column summarizing the impact.

Limitations and Future Work

  • Dataset Sensitivity: The authors observe that the effectiveness of certain perturbations can vary across datasets, and performance on TrecQA showed significant variance.
  • Future Directions: The paper suggests exploring task-related constraints, such as incorporating word sentiment polarity or distances to entities. A more ambitious future direction is to revisit the second strategy mentioned in the introduction: changing text using additional knowledge (e.g., leveraging knowledge bases or language models for token replacement), which is more challenging but potentially more powerful.

Summary updated

Abstract

In this technique report, we aim to mitigate the overfitting problem of natural language by applying data augmentation methods. Specifically, we attempt several types of noise to perturb the input word embedding, such as Gaussian noise, Bernoulli noise, and adversarial noise, etc. We also apply several constraints on different types of noise. By implementing these proposed data augmentation methods, the baseline models can gain improvements on several sentence classification tasks.
BibTeX
@misc{zhang2018word,
  author = {Dongxu Zhang and Zhichao Yang},
  title = {{Word Embedding Perturbation for Sentence Classification}},
  year = {2018},
  eprint = {1804.08166},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.08166},
}
Figure