Defending Against Unforeseen Failure Modes with Latent Adversarial Training

Stephen Casper, Lennart Schulze, Oam Patel, Dylan Hadfield-Menell

Introduction

Machine learning models often exhibit vulnerabilities that developers fail to anticipate during training. While standard Adversarial Training (AT) is effective at mitigating known failure modes by training on specific adversarial examples, it remains brittle against unforeseen "black swan" events or hidden backdoors. The core challenge lies in the vastness of the input space; it is computationally impossible to exhaustively search for every potential trigger.

Overview of the LAT framework: (a) the problem of unforeseen failure modes, (b) motivation for latent attacks, (c) training methodology, and (d) evaluation setup.

The Latent Adversarial Training (LAT) Framework

To address these "unknown unknowns," this research introduces Latent Adversarial Training (LAT). Unlike standard AT, which perturbs raw inputs (e.g., pixels or text tokens), LAT targets a model's internal latent representations.

Methodology

In standard AT, an input $x$ is modified by a perturbation $\delta_x$ to maximize loss, and the model is trained to minimize this loss. In contrast, LAT feeds a clean input into the network but applies a perturbation $\delta_\ell$ directly to the activations of a hidden layer. By using Projected Gradient Descent (PGD) to find internal "tweaks" that maximize loss, LAT forces the model to develop robust internal circuitry. This approach leverages the fact that latent spaces are more compressed and structured, making it easier to elicit failure modes without the adversary needing to know the specific input-space triggers.

Experimental Evaluation

The effectiveness of LAT was evaluated across three distinct domains to demonstrate its versatility:

  1. Image Classification: Using ImageNet poisoned with eight different backdoor types, including both patch and natural feature triggers.
  2. Text Classification: Using the Redwood injurious text dataset to identify descriptions of harm, featuring eight implanted backdoors.
  3. Text Generation: Using Llama-2-7b-chat on the Anthropic Helpful-Harmless (HH) and PKU BeaverTails datasets to scrub harmful behaviors and nonsensical triggers.

Key Insights

Expanding the Pareto Frontier

LAT consistently provided a superior trade-off between clean accuracy and robust performance compared to standard AT. While standard AT often significantly degrades performance on normal data, LAT maintains high clean accuracy while providing broader protection against novel, unseen attacks.

Layer Sensitivity and Efficiency

The choice of the target layer for perturbation is critical. Experiments indicated that attacking early layers (e.g., Layer 3 in a 24-layer transformer) is most effective, as it allows subsequent layers to "correct" the corrupted signal. Furthermore, LAT is computationally more efficient than standard AT; because perturbations start at a hidden layer, it requires fewer operations than a full forward/backward pass through the entire network.

Results and Performance

  • Robustness Gains: In image classification tasks, LAT achieved a 68% greater improvement in the Area Under the Curve (AUC) for robustness against novel, unseen attacks compared to standard AT.
  • Backdoor Scrubbing: LAT proved highly effective at removing hidden backdoors in text generation models, significantly outperforming embedding-space AT in the Anthropic-HH tasks.
  • Generalization: By training against adversarial latent states, the models learned to rely on high-level, robust features rather than brittle input-level patterns.

Limitations and Future Directions

Despite its advantages, LAT requires careful hyperparameter tuning, particularly regarding the choice of layer and the magnitude of the perturbation $\epsilon$. Unlike the input space, where "small" changes are visually or semantically intuitive, interpreting the scale of latent perturbations is challenging. Additionally, while LAT excelled in most scenarios, standard AT performed better in specific cases like the BeaverTails dataset, where triggers were more easily identifiable in the input space.

Future research will explore targeted LAT, where internal adversaries aim for specific harmful outputs, and evaluate the scalability of this approach on larger frontier models like GPT-4 and Llama-3.

Summary updated

BibTeX
@article{casper2025defending,
  author = {Stephen Casper and Lennart Schulze and Oam Patel and Dylan Hadfield-Menell},
  title = {{Defending Against Unforeseen Failure Modes with Latent Adversarial Training}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=mVPPhQ8cAd},
}
Figure