Meta Style Adversarial Training for Cross-Domain Few-Shot Learning

Yuqian Fu, Yu Xie, Yanwei Fu, Yu-Gang Jiang

Abstract

Cross-Domain Few-Shot Learning (CD-FSL) aims to transfer knowledge from a source domain to novel target domains with limited data. A primary challenge in CD-FSL is the significant domain gap caused by shifts in visual styles. While existing methods like wave-SAN attempt to bridge this gap by swapping styles between images, they are limited to "real" and "easy" styles already present in the source data. This paper proposes StyleAdv, a model-agnostic meta-learning framework that synthesizes "virtual" and "hard" adversarial styles. By perturbing style statistics with signed gradients, StyleAdv forces the model to become robust against drastic visual shifts. Extensive experiments on eight target datasets demonstrate that StyleAdv achieves state-of-the-art performance using both CNN and Vision Transformer (ViT) backbones.

The Challenge of Domain Gaps

Few-Shot Learning (FSL) typically assumes that training and test data come from the same distribution. In contrast, CD-FSL involves moving from general domains (e.g., ImageNet) to specialized ones like medical X-rays or satellite imagery. The "style" of an image—its texture, lighting, and color distribution—is the primary driver of this domain gap. To build a robust model, we must move beyond simple style-swapping and create synthetic styles that specifically challenge the model's current feature representations.

Methodology: The StyleAdv Framework

StyleAdv operates as a minimax game within a meta-learning loop, consisting of an inner loop for style attack and an outer loop for model optimization.

The overall framework of StyleAdv (left) and its style extraction modules for CNN and ViT backbones (right).

1. Style Extraction

Inspired by Adaptive Instance Normalization (AdaIN), the model extracts the mean ($\mu$) and standard deviation ($\sigma$) of feature maps. For CNNs, these are calculated across spatial dimensions. For Vision Transformers, the framework reshapes patch tokens into spatial feature maps to compute equivalent channel-wise statistics.

2. Inner Loop: Style Adversarial Attack

The core innovation is a novel Style-FGSM (Fast Gradient Sign Method). Instead of perturbing raw pixels, it adds noise to the extracted style parameters ($\mu, \sigma$) in the direction that increases classification loss. This process creates "hard" adversarial styles that do not exist in the training set but are mathematically designed to expose model weaknesses.

3. Outer Loop: Optimization

The model is trained to minimize a multi-part loss function:

  • Standard FSL Loss: Accuracy on clean images.
  • Adversarial Loss: Accuracy on images with synthesized hard styles.
  • Consistency Loss: A KL-Divergence term that ensures the model's predictions remain stable even when the style changes, preventing "semantic drift."

Key Results and Insights

StyleAdv was evaluated on the BSCD-FSL and FWT benchmarks, covering eight diverse target domains. It achieved new state-of-the-art results, improving ResNet-10 base models by up to 8.44% in 5-way 5-shot settings.

Visual evidence of StyleAdv: (a) synthesized adversarial styles, (b) loss curves showing increased meta-task difficulty, and (c) T-SNE visualization of feature clusters.

As illustrated in Figure 6, the adversarial styles (a) significantly alter the visual appearance while preserving the underlying object structure. The training curves (b) confirm that adversarial styles are indeed "harder" to learn, leading to higher loss during training but ultimately better generalization. T-SNE visualizations (c) show that StyleAdv produces much tighter and more separable class clusters compared to baseline methods, which is critical for few-shot classification under heavy domain shift.

Conclusion

StyleAdv demonstrates that attacking the latent style space is more effective for cross-domain generalization than attacking raw pixels. By forcing the model to focus on core semantic features rather than superficial style cues, StyleAdv provides a robust solution for CD-FSL across both traditional and transformer-based architectures.

Summary updated

Abstract

Cross-Domain Few-Shot Learning (CD-FSL) is a recently emerging task that tackles few-shot learning across different domains. It aims at transferring prior knowledge learned on the source dataset to novel target datasets. The CD-FSL task is especially challenged by the huge domain gap between different datasets. Critically, such a domain gap actually comes from the changes of visual styles, and wave-SAN empirically shows that spanning the style distribution of the source data helps alleviate this issue. However, wave-SAN simply swaps styles of two images. Such a vanilla operation makes the generated styles ``real'' and ``easy'', which still fall into the original set of the source styles. Thus, inspired by vanilla adversarial learning, a novel model-agnostic meta Style Adversarial training (StyleAdv) method together with a novel style adversarial attack method is proposed for CD-FSL. Particularly, our style attack method synthesizes both ``virtual'' and ``hard'' adversarial styles for model training. This is achieved by perturbing the original style with the signed style gradients. By continually attacking styles and forcing the model to recognize these challenging adversarial styles, our model is gradually robust to the visual styles, thus boosting the generalization ability for novel target datasets. Besides the typical CNN-based backbone, we also employ our StyleAdv method on large-scale pretrained vision transformer. Extensive experiments conducted on eight various target datasets show the effectiveness of our method. Whether built upon ResNet or ViT, we achieve the new state of the art for CD-FSL. Code is available at https://github.com/lovelyqian/StyleAdv-CDFSL.
BibTeX
@inproceedings{fu2023meta,
  author = {Yuqian Fu and Yu Xie and Yanwei Fu and Yu-Gang Jiang},
  title = {{Meta Style Adversarial Training for Cross-Domain Few-Shot Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02354},
}
Figure