Adversarial Semantic Collisions

Congzheng Song, Alexander M. Rush, Vitaly Shmatikov

Abstract

Modern Natural Language Processing (NLP) models often rely on internal representations that are "excessively invariant," meaning they ignore significant portions of the input signal. This paper introduces semantic collisions, a new class of vulnerabilities where unrelated or nonsensical text strings are mistakenly judged as semantically identical to a target query. Using a gradient-based optimization framework, we demonstrate that state-of-the-art models for retrieval, summarization, and chatbots can be manipulated with near-perfect success rates. Our findings reveal a fundamental flaw in how these models represent meaning, allowing wildly different texts to "collide" in the model's internal representation space.

Introduction

Most adversarial research in NLP focuses on "adversarial examples"—small, often imperceptible changes to a sentence that trick a model into changing its output. This work explores the inverse problem: Semantic Collisions. Instead of similar inputs producing different outputs, these are dissimilar inputs that produce the same model output. For instance, a search engine using a neural model might find a piece of gibberish to be a "perfect match" for a user's query. This exposure highlights a critical weakness: models are often too invariant to changes that should fundamentally alter semantic meaning.

Methodology

To generate these collisions, we developed a gradient-based framework assuming "white-box" access to the victim model's weights. Since text is discrete, we employ a continuous relaxation technique followed by a discrete projection.

  1. Soft Optimization: Words are represented as continuous vectors (logits). We use gradient ascent to maximize the similarity score between the target query and the generated text.
  2. Discrete Projection: A specialized beam search is used to map these continuous vectors back into actual tokens that maintain high similarity scores according to the model.
Overview of the collision generation framework using continuous relaxation and beam search.

We proposed three attack variants:

  • Aggressive: Unconstrained strings that maximize similarity scores, often appearing as gibberish.
  • Regularized Aggressive: Incorporates a penalty for high perplexity to reduce the "chaos" of the generated text.
  • Natural: Employs a joint search with a pre-trained language model (e.g., GPT-2) to generate fluent, human-readable sentences that still function as collisions.

Experimental Setup

The attacks were evaluated across four major NLP tasks using standard benchmarks:

  • Paraphrase Identification: MRPC and QQP corpora.
  • Document Retrieval: TREC Common Core Tracks (Core17/18).
  • Response Suggestion: Persona-chat dialogue contexts.
  • Extractive Summarization: CNN/DailyMail (CNNDM).

Results and Key Insights

The attacks proved devastatingly effective. In document retrieval, inserting a collision into an irrelevant document shifted its median rank from 1000th to the top 10. In dialogue systems, "Aggressive" attacks were ranked as the top response 99% of the time, and extractive summarization models chose the collision as the summary 100% of the time.

Distribution of similarity scores across different datasets, comparing real data to various collision types.

Key findings include:

  • Model Over-Confidence: Models often identify nonsensical collisions as paraphrases with higher confidence (>99%) than actual human-written paraphrases.
  • Evading Defenses: While "perplexity filtering" is a common defense against gibberish, our "Natural" collisions produce fluent English that bypasses these filters while remaining effective.
  • Transferability: Collisions generated for one BERT-based model often transfer to others, suggesting the vulnerability is inherent to the architecture or pre-training process.

Conclusion and Future Work

While the current attack requires white-box access, the high success rates and transferability indicate a significant security risk. Future work should explore adversarial training—incorporating collisions into the training set—to improve model sensitivity to semantic differences. There is a pressing need for more robust "meaning representations" that do not suffer from excessive invariance.

Summary updated

Abstract

We study semantic collisions: texts that are semantically unrelated but judged as similar by NLP models. We develop gradient-based approaches for generating semantic collisions and demonstrate that state-of-the-art models for many tasks which rely on analyzing the meaning and similarity of texts—including paraphrase identification, document retrieval, response suggestion, and extractive summarization—are vulnerable to semantic collisions. For example, given a target query, inserting a crafted collision into an irrelevant document can shift its retrieval rank from 1000 to top 3. We show how to generate semantic collisions that evade perplexity-based filtering and discuss other potential mitigations. Our code is available at https://github.com/csong27/collision-bert.
BibTeX
@inproceedings{song2020adversarial,
  author = {Congzheng Song and Alexander M. Rush and Vitaly Shmatikov},
  title = {{Adversarial Semantic Collisions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.344},
}
Figure