Introduction
Adversarial attacks on Natural Language Processing (NLP) models typically fall into two categories: white-box, where the attacker has full access to the model, and black-box, where the attacker can observe model outputs (labels or scores). However, real-world security often presents a "hard black-box" scenario: the target model is completely hidden, and excessive querying to probe for weaknesses would trigger security alarms. Q-FAKER addresses this gap by introducing a query-free attack method that generates high-quality, fluent adversarial examples without any interaction with the target model.
Methodology: Controlled Generation and Adversarial Steering
The core innovation of Q-FAKER is its three-step process designed to "steer" a generative language model (like GPT-2) toward adversarial outputs using a surrogate model.
- Surrogate Model Training: Since the target is inaccessible, a small surrogate model (a single-layer classification head on GPT-2) is trained on a different dataset for the same task. This relies on the insight that adversarial features for specific tasks (like spam or toxicity) tend to be universal across architectures.
- Adversarial Steering: The system calculates "adversarial gradients" from the surrogate, identifying how internal representations should shift to cause misclassification.
- Controlled Generation: Using a technique called post-norm fusion, these gradients shift the generator's hidden states. This ensures the output remains fluent English while simultaneously embedding adversarial features that transfer to the victim model.
Experimental Evaluation
Q-FAKER was evaluated using the Advbench benchmark across four critical tasks: Spam Detection (Enron, Assassin), Sensitive Information Detection (EDENCE, FAS), Disinformation Detection (CGFake, Amazon-LB), and Toxicity Detection (Jigsaw, HSOL). The attacks were tested against eight victim models, ranging from BERT and RoBERTa to large language models (LLMs) like LLaMA-3 and Mistral-7B.
Key Results and Analysis
Q-FAKER demonstrates significant improvements over existing state-of-the-art methods, particularly in restricted query environments.
Zero-Query Dominance
In the "Sensitive Information" task (EDENCE dataset), Q-FAKER achieved a 54.6% Attack Success Rate (ASR) with zero queries. In contrast, traditional black-box methods like TextFooler or BERT-Attack fail completely (0% ASR) when queries are restricted to one, as they rely on iterative probing. Q-FAKER maintains its performance regardless of the query limit because it generates the adversarial sentence in a single forward pass.
High Transferability and Quality
The attack exhibits remarkable transferability; adversarial examples generated against a BERT-base surrogate successfully fooled diverse architectures, including DeBERTa and even zero-shot prompted LLMs. Furthermore, human evaluations confirmed that Q-FAKER's outputs are more natural and grammatically correct than those produced by word-substitution attacks, making them harder for both humans and automated detectors to spot.
Conclusion
Q-FAKER proves that task-specific knowledge is often more critical than model-specific knowledge for successful adversarial attacks. By leveraging controlled generation, it bypasses the need for costly queries and extensive training data (requiring only 1,000 trainable parameters compared to millions in previous methods). While it requires knowledge of the target's task, its efficiency and stealth pose a significant new challenge for NLP security. Future work will explore iterative versions to further boost success rates in scenarios where limited querying is permissible.
Summary updated