Abstract
Modern Natural Language Processing (NLP) models often rely on internal representations that are "excessively invariant," meaning they ignore significant portions of the input signal. This paper introduces semantic collisions, a new class of vulnerabilities where unrelated or nonsensical text strings are mistakenly judged as semantically identical to a target query. Using a gradient-based optimization framework, we demonstrate that state-of-the-art models for retrieval, summarization, and chatbots can be manipulated with near-perfect success rates. Our findings reveal a fundamental flaw in how these models represent meaning, allowing wildly different texts to "collide" in the model's internal representation space.
Introduction
Most adversarial research in NLP focuses on "adversarial examples"—small, often imperceptible changes to a sentence that trick a model into changing its output. This work explores the inverse problem: Semantic Collisions. Instead of similar inputs producing different outputs, these are dissimilar inputs that produce the same model output. For instance, a search engine using a neural model might find a piece of gibberish to be a "perfect match" for a user's query. This exposure highlights a critical weakness: models are often too invariant to changes that should fundamentally alter semantic meaning.
Methodology
To generate these collisions, we developed a gradient-based framework assuming "white-box" access to the victim model's weights. Since text is discrete, we employ a continuous relaxation technique followed by a discrete projection.
- Soft Optimization: Words are represented as continuous vectors (logits). We use gradient ascent to maximize the similarity score between the target query and the generated text.
- Discrete Projection: A specialized beam search is used to map these continuous vectors back into actual tokens that maintain high similarity scores according to the model.
We proposed three attack variants:
- Aggressive: Unconstrained strings that maximize similarity scores, often appearing as gibberish.
- Regularized Aggressive: Incorporates a penalty for high perplexity to reduce the "chaos" of the generated text.
- Natural: Employs a joint search with a pre-trained language model (e.g., GPT-2) to generate fluent, human-readable sentences that still function as collisions.
Experimental Setup
The attacks were evaluated across four major NLP tasks using standard benchmarks:
- Paraphrase Identification: MRPC and QQP corpora.
- Document Retrieval: TREC Common Core Tracks (Core17/18).
- Response Suggestion: Persona-chat dialogue contexts.
- Extractive Summarization: CNN/DailyMail (CNNDM).
Results and Key Insights
The attacks proved devastatingly effective. In document retrieval, inserting a collision into an irrelevant document shifted its median rank from 1000th to the top 10. In dialogue systems, "Aggressive" attacks were ranked as the top response 99% of the time, and extractive summarization models chose the collision as the summary 100% of the time.
Key findings include:
- Model Over-Confidence: Models often identify nonsensical collisions as paraphrases with higher confidence (>99%) than actual human-written paraphrases.
- Evading Defenses: While "perplexity filtering" is a common defense against gibberish, our "Natural" collisions produce fluent English that bypasses these filters while remaining effective.
- Transferability: Collisions generated for one BERT-based model often transfer to others, suggesting the vulnerability is inherent to the architecture or pre-training process.
Conclusion and Future Work
While the current attack requires white-box access, the high success rates and transferability indicate a significant security risk. Future work should explore adversarial training—incorporating collisions into the training set—to improve model sensitivity to semantic differences. There is a pressing need for more robust "meaning representations" that do not suffer from excessive invariance.
Summary updated