Abstract
Zero-Shot Learning (ZSL) models are notoriously fragile against adversarial attacks because they rely entirely on transferring knowledge to unseen classes without direct training data. This paper introduces ATZSL, an adversarially trained ZSL framework that uses a novel defensive network architecture to maintain high accuracy on clean data while drastically improving robustness against both image and semantic attribute attacks.
Introduction
Zero-Shot Learning (ZSL) aims to recognize object classes that a model has never seen during training. It achieves this by relying on auxiliary semantic information, such as a list of attributes (e.g., "has stripes," "four legs," "hooves" to identify a zebra). While state-of-the-art neural networks are already vulnerable to adversarial attacks—tiny, invisible perturbations that trick the model into making wrong predictions—ZSL models are uniquely susceptible.
Because ZSL models lack training data for unseen classes, they rely heavily on a single semantic prototype (attribute vector) per class to bridge the gap. If an attacker slightly perturbs the input image, or subtly poisons the semantic description, the fragile visual-semantic mapping breaks down entirely. Despite this glaring security risk, prior ZSL research has focused almost exclusively on accuracy and transferability, leaving adversarial robustness largely ignored.
Methodology
To tackle this, the authors propose ATZSL (Adversarially Trained Zero-Shot Learning). The approach introduces two major innovations:
1. Adversarial Training via Min-Max Optimization
ATZSL injects adversarial examples into the training process. It frames the training as a min-max game: the inner loop generates worst-case adversarial perturbations (maximizing the model's loss), while the outer loop updates the model weights to correctly classify these perturbed examples (minimizing the loss). The framework is flexible enough to generate and defend against attacks in both the visual space (image pixels) and the semantic space (class attributes).
2. Defensive Relation Prediction Network
Traditional ZSL models often project visual and semantic features into a shared space and use a dot product to measure similarity. The authors note that this parallel projection and multiplication actually amplifies small adversarial perturbations. Instead, ATZSL extracts the visual and semantic embeddings, concatenates them, and feeds the combined vector into a Multi-Layer Perceptron (MLP) to predict a relation score. This structural change acts as a natural defense mechanism.
Experimental Setup
The authors evaluated their model on two standard benchmark datasets for ZSL, using rigorous splits to ensure no overlap with ImageNet pre-training:
- AWA2 (Animals with Attributes 2): A coarse-grained dataset containing 50 animal classes and 85 attributes.
- CUB (CUB-200-2011 Birds): A fine-grained dataset containing 200 bird species annotated with 312 attributes.
Key Insights and Results
Robustness Transfer
A major finding is that adversarial training on seen classes successfully transfers defensive capabilities to unseen classes. This demonstrates that the model learns a robust visual-semantic mapping that generalizes beyond the training distribution.
Performance Gains
The authors tested ATZSL against a battery of standard white-box and black-box attacks (FGSM, IFGSM, DeepFool, CW, WRM, ZOO) under both Standard ZSL and Generalized ZSL settings.
- Image Attacks: On the AWA2 dataset, ATZSL improved the harmonic mean of clean and adversarial accuracy by 7.5% to 19.7% over the strongest competitor.
- Semantic Attacks: When defending against attribute attacks on the CUB dataset, ATZSL improved the trade-off performance by 14.6% to 26.8%.
- Clean Accuracy: Crucially, ATZSL achieved these boosts with only a negligible drop in accuracy on clean, unperturbed images.
Conclusion
ATZSL sets a strong new baseline for robust ZSL by addressing the inherent fragility of visual-semantic mappings. By shifting from dot-product similarity to a concatenation-based relation network and employing adversarial training, the model achieves state-of-the-art defensive performance. While a gap remains between clean and adversarial accuracy on unseen classes, this work highlights a critical security dimension for future ZSL research.
Summary updated