EFSG: Evolutionary Fooling Sentences Generator

Marco Di Giovanni, Marco Brambilla

Introduction

Large Language Models (LLMs) like BERT and RoBERTa have set new benchmarks in Natural Language Processing (NLP). However, their internal mechanisms remain largely opaque. This research investigates whether these transformer-based models possess vulnerabilities similar to the "white noise" susceptibility found in computer vision, where random patterns can trigger high-confidence misclassifications. The authors introduce the Evolutionary Fooling Sentences Generator (EFSG), a black-box adversarial attack designed to expose these weaknesses using Darwinian principles.

Methodology

EFSG is a model-agnostic and task-agnostic evolutionary algorithm. It treats the target model's confidence score for a positive classification as a "fitness" metric. The process begins with an initial population of random word sequences sampled from a training vocabulary. Through successive generations, the algorithm applies selection, crossover, and mutation:

  1. Selection: Sentences with the highest confidence scores are chosen for reproduction.
  2. Crossover: Features from successful adversarial examples are combined to explore new regions of the input space.
  3. Mutation: Random words are swapped to maintain diversity.

This iterative process continues until the algorithm generates "fooling sentences"—nonsense strings that the model classifies as valid English or correct paraphrases with near-total certainty. The inclusion of crossover was found to be critical for overcoming performance plateaus, suggesting that models are sensitive to specific combinations of linguistic features.

Evolutionary progress of fooling sentences (M) over epochs for CoLA and MRPC at various confidence thresholds (T).

Experimental Setup

The study evaluates EFSG against BERT (Base and Large) and RoBERTa on two binary classification tasks from the GLUE benchmark:

  • CoLA (Corpus of Linguistic Acceptability): Assessing grammatical correctness.
  • MRPC (Microsoft Research Paraphrase Corpus): Identifying semantic equivalence between sentence pairs.

The vocabulary for the generator is derived from the training sets of these corpora to ensure the models are familiar with the individual tokens.

Results and Discussion

The experiments demonstrate that LLMs are indeed susceptible to "textual white noise." EFSG successfully generated gibberish strings that models rated as highly grammatical or semantically equivalent. As shown in the evolutionary plots, the number of successful fooling sentences ($M$) increases significantly over generations, particularly for lower confidence thresholds ($T$).

Evaluation accuracy of different models on CoLA and MRPC tasks, including the "BERT improved" version trained with adversarial examples.

While RoBERTa showed greater inherent robustness than BERT, it remained vulnerable. To address this, the authors implemented an adversarial training defense. By incorporating the generated fooling sentences into the training set with negative labels, they produced a "BERT improved" model. This version exhibited significantly higher robustness against EFSG attacks while maintaining its original performance on standard benchmarks. Furthermore, the research found that vulnerabilities are often shared: if one fine-tuning of BERT is fooled by a specific nonsense string, there is a 40-60% chance that a different fine-tuning of the same model will also be fooled.

Conclusion

The EFSG framework highlights a significant gap between a model's confidence and its actual linguistic competence. By exposing these "white noise" vulnerabilities, the research provides a pathway toward more interpretable and robust NLP systems. Future work will focus on extending EFSG to non-binary tasks and incorporating constraints to generate more human-readable adversarial examples.

Summary updated

Abstract

Large pre-trained language representation models (LMs) have recently collected a huge number of successes in many NLP tasks. In 2018 BERT, and later its successors (e.g. RoBERTa), obtained state-of-the-art results in classical benchmark tasks, such as GLUE benchmark. After that, works about adversarial attacks have been published to test their generalization proprieties and robustness. In this work, we design Evolutionary Fooling Sentences Generator (EFSG), a model- and task-agnostic adversarial attack algorithm built using an evolutionary approach to generate false-positive sentences for binary classification tasks. We successfully apply EFSG to CoLA and MRPC tasks, on BERT and RoBERTa, comparing performances. Results prove the presence of weak spots in state-of-the-art LMs. We finally test adversarial training as a data augmentation defence approach against EFSG, obtaining stronger improved models with no loss of accuracy when tested on the original datasets.
BibTeX
@inproceedings{giovanni2020efsg,
  author = {Marco Di Giovanni and Marco Brambilla},
  title = {{EFSG: Evolutionary Fooling Sentences Generator}},
  booktitle = {Icsc},
  year = {2020},
  doi = {10.1109/icsc50631.2021.00037},
}
Figure