Abstract
Adversarial attacks on Natural Language Processing (NLP) models typically require either internal model access or extensive querying of APIs. This paper presents VDBA (Victim Data-based Adversarial Attack), a formidable new paradigm that requires neither. By utilizing a small set of unlabeled "victim texts" to build a hierarchical suite of substitute models, VDBA achieves high attack success rates against state-of-the-art systems like GPT-4o and Qwen2 with zero initial queries.
Introduction
Adversarial attacks—subtly modifying input text to trick a model into misclassification—pose a significant security risk. However, most existing methods are impractical for real-world deployment. White-box attacks require full access to model weights, while query-based black-box attacks are often thwarted by rate limits and detection mechanisms.
VDBA addresses this gap by demonstrating that an attacker only needs "victim texts"—unlabeled data similar to what the target model processes—to craft successful exploits. This "query-less" approach allows attackers to remain completely invisible until the moment of the attack.
Methodology
The VDBA framework operates through a three-stage pipeline designed to simulate the victim model's decision boundaries without direct interaction.
1. Cluster-Guided Shadow Dataset Synthesis
In the absence of labeled data, VDBA utilizes unlabeled victim texts. These are transformed into vectors using a pre-trained embedding model (e.g., T5). K-means clustering is then applied to partition the data, assigning "pseudo-labels" to create a "shadow dataset" that approximates the target's task structure.
2. Hierarchical Substitute Model Design
To capture the nuances of a victim model's decision boundary, VDBA employs a "divide and conquer" strategy. It iteratively re-clusters data to train a hierarchy of substitute models. This ensures that even if one model is inaccurate, others in the hierarchy likely mirror the victim's logic for specific text subsets.
3. Diverse Adversarial Example Generation
VDBA employs a "committee" of established attack methods (such as BAE, Hotflip, and PSO) across its substitute models. A scoring function selects the optimal adversarial candidate that maximizes attack effectiveness while maintaining high semantic similarity to the original input.
Experimental Evaluation
The authors evaluated VDBA across several standard NLP benchmarks, including SST5 and Emotion for sentiment analysis, and AG News and TREC6 for classification tasks. To test extreme scenarios, "one-shot" sources were pulled from Go-Emotions and Rotten Tomatoes to train shadow models with minimal initial data.
Key Results
- State-of-the-Art Performance: On the SST5 dataset, VDBA achieved an Attack Success Rate (ASR) of 52.08%, a significant improvement over traditional transfer-based methods, despite requiring zero queries.
- LLM Vulnerability: The attack proved remarkably effective against closed-source models, achieving an ASR of 45.99% on Qwen2 and over 35% on GPT-4o.
- Data Scarcity Resilience: Even with as few as five victim texts, VDBA maintained an ASR of over 27%, highlighting the risk posed by even minor data leaks.
Discussion and Conclusion
The core insight of VDBA is that exact label replication is unnecessary; simulating high-level decision boundaries through clustering is sufficient for successful transfer attacks. Furthermore, the attack's effectiveness with simple "one-hot" encodings suggests it does not require sophisticated hardware.
While highly effective, VDBA faces a trade-off between the number of hierarchical iterations and semantic similarity. Aggressive attacks can occasionally result in less natural text. Future research will explore extending VDBA to other modalities like image or audio and developing more robust defense mechanisms against data-based exploits.
Summary updated