A Sweet Rabbit Hole by DARCY: Using Honeypots to Detect Universal Trigger's Adversarial Attacks

Thai Le, Noseong Park, Dongwon Lee

This paper introduces DARCY (Defending universAl tRigger’s attaCk with honeYpot), a novel framework designed to protect NLP models from one of their most potent threats: the Universal Trigger attack. By borrowing the "honeypot" concept from cybersecurity, the authors demonstrate how to bait adversarial algorithms into a "sweet rabbit hole," making attacks easy to detect without compromising the model's original performance.

TL;DR

DARCY defends NLP models against Universal Trigger attacks by greedily injecting multiple "trapdoors" (honeypots) into the neural network's loss landscape. This forces adversarial optimization to converge on predictable, pre-defined triggers, allowing for detection rates of up to 99% with minimal impact on clean-text accuracy.

What Problem Does This Paper Address?

Adversarial examples in NLP typically involve small, intentional perturbations to text that fool a machine learning model. While many attacks are instance-specific, the Universal Trigger (UniTrigger) attack is particularly dangerous. It "learns" a short, fixed phrase (e.g., "zoning zoombie") that, when added to any input, causes the model to output a specific target class with near-certainty.

Because UniTrigger is learning-based, it generalizes across unseen inputs and is highly transferable between different model architectures. Existing defenses, such as checking for semantic similarity or using robust word recognizers, often fail to stop these powerful, low-cost "mass attacks."

Methods

The DARCY framework operates in three main stages:

  1. Searching Trapdoors: The system uses a Multiple Greedy Trapdoor Search algorithm to find tokens that can act as effective baits. These tokens are selected based on three criteria:
  2. Fidelity: They shouldn't contradict the semantic meaning of the target label.
  3. Robustness: Multiple trapdoors ($K \ge 1$) are found to prevent an attacker from simply bypassing a single known bait.
  4. Class-Awareness: Trapdoors for different labels are kept far apart in the embedding space to avoid confusing the model.
  5. Injecting Trapdoors: The target model is re-trained to associate these trapdoor tokens with specific labels. For example, if "queen gambit" is a trapdoor for the "Positive" class, the model is trained to classify any negative review containing that phrase as "Positive." This creates artificial local optima in the model's loss landscape.
  6. Detecting Trapdoors: A separate binary classifier is trained to monitor the model's internal representations. It learns to distinguish between normal inputs and those containing the injected trapdoors.
Figure 1: An example of DARCY. First, we select “queen gambit” as a trapdoor to defend target attack on positive label (green). Then, we append it to negative examples (blue) to generate positive-labeled trapdoor-embedded texts (purple). Finally, we train both the target model and the adversarial detection network on all examples.

Datasets

The authors evaluated DARCY across four diverse public datasets:

  • Subjectivity (SJ): Classifying sentences as subjective or objective.
  • Movie Reviews (MR): Binary sentiment analysis.
  • Binary Sentiment Treebank (SST): Sentiment analysis on movie reviews.
  • AG News: A 4-class news topic classification dataset.

Key Insights and Ideas

  • Baiting the Optimizer: UniTrigger relies on an optimization process to find triggers. DARCY exploits this by creating "easier" optimization targets (the trapdoors) that the attacker's algorithm will naturally gravitate toward.
  • The Power of Multiplicity: A single trapdoor is vulnerable to an adaptive attacker who might learn to avoid it. By injecting multiple trapdoors, DARCY significantly increases the "surface area" of the defense, making it much harder for an attacker to find a trigger that doesn't trip a honeypot.
  • Greedy vs. Random: The paper shows that greedily searching for trapdoors based on the model's feature space is far more effective than random selection, ensuring the trapdoors are "strong" enough to attract the attack optimization.
Figure 2: Multiple Greedy Trapdoor Search

Key Results

  • Near-Perfect Detection: In "Novice" attack scenarios (where the attacker doesn't know about the trapdoors), DARCY achieved a detection AUC of 99% in most cases, with a True Positive Rate (TPR) of up to 99% and a False Positive Rate (FPR) below 2%.
  • Preserved Utility: The F1 score for clean, non-adversarial inputs remained within a 1% margin of the original undefended model.
  • Resilience to Adaptive Attacks: Even when attackers were aware of the defense and tried to avoid the trapdoors, DARCY (with $K=5$ or more) remained robust, maintaining high detection rates where other baselines failed.
Figure 3: DARCY and SelfATK under novice attack

Limitations and Future Work

  • Oracle Attacks: The defense is most vulnerable to an "Oracle" attacker who has full access to the detection network. In this case, the attacker can explicitly optimize to avoid the detector, significantly reducing DARCY's effectiveness.
  • Generalization: While highly effective against UniTrigger, the current implementation is specific to that attack style. The authors plan to extend DARCY to safeguard against other types of NLP adversarial generators in the future.
  • Black-Box Transferability: The transferability of trapdoors from a surrogate model to a black-box target model showed mixed results depending on the dataset, suggesting a need for more robust model-stealing or transfer techniques.

Summary updated

Abstract

The Universal Trigger (UniTrigger) is a recently-proposed powerful adversarial textual attack method. Utilizing a learning-based mechanism, UniTrigger generates a fixed phrase that, when added to any benign inputs, can drop the prediction accuracy of a textual neural network (NN) model to near zero on a target class. To defend against this attack that can cause significant harm, in this paper, we borrow the "honeypot" concept from the cybersecurity community and propose DARCY, a honeypot-based defense framework against UniTrigger. DARCY greedily searches and injects multiple trapdoors into an NN model to "bait and catch" potential attacks. Through comprehensive experiments across four public datasets, we show that DARCY detects UniTrigger's adversarial attacks with up to 99% TPR and less than 2% FPR in most cases, while maintaining the prediction accuracy (in F1) for clean inputs within a 1% margin. We also demonstrate that DARCY with multiple trapdoors is also robust to a diverse set of attack scenarios with attackers' varying levels of knowledge and skills. Source code will be released upon the acceptance of this paper.
BibTeX
@inproceedings{le2021sweet,
  author = {Thai Le and Noseong Park and Dongwon Lee},
  title = {{A Sweet Rabbit Hole by DARCY: Using Honeypots to Detect Universal Trigger's Adversarial Attacks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.296},
}
Figure