Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation

AprilPyone MaungMaung, Huy H. Nguyen, Hitoshi Kiya, Isao Echizen

Abstract

Deep Neural Networks (DNNs) often rely on "shortcut learning," where they associate spurious features—such as backgrounds or co-occurring objects—with specific classes rather than learning the core object characteristics. This paper introduces a method to fine-tune text-to-image diffusion models to generate an unlimited supply of these spurious images. By incorporating a novel Spurious Feature Similarity Loss (SFSL), the researchers demonstrate that synthetic images can be crafted to "universally" trick various model architectures, including ResNets and Vision Transformers. This approach provides a robust framework for auditing and debugging computer vision systems against real-world failures.

Introduction

Deep learning models are notoriously prone to learning spurious correlations. For instance, a model might identify a "hummingbird" based on the presence of red tubular flowers rather than the bird itself.

A real hummingbird, representing the target class for which spurious features are often learned.

While datasets like Spurious ImageNet provide examples of these failures, they suffer from data scarcity and a lack of universality—features that trick one model may not trick another. This research addresses these limitations by using generative AI to create a "synthetic laboratory" for spurious feature analysis.

Methodology

The authors leverage DreamBooth to fine-tune Stable Diffusion on a small set of reference spurious images. To ensure the generated images prioritize the misleading "shortcuts" over general visual fidelity, they introduce the Spurious Feature Similarity Loss (SFSL).

This loss function utilizes an adversarially robust model as a feature extractor, calculating the cosine similarity between the neural representations of generated and reference images. This process forces the diffusion model to capture the specific textures and shapes (e.g., flower patterns or flagpole structures) that characterize the spurious class.

Comparison between original Spurious ImageNet references and the images generated using the proposed method.

Key Results and Insights

The generated images achieved significantly higher "spurious accuracy"—the rate at which classifiers are successfully misled—compared to the original dataset. For example, in the "Gondola" class, the synthetic images reached 100% spurious accuracy across four different architectures (ResNet-50 v1/v2, Robust ResNet, and ViT). Ablation studies confirmed that the SFSL was critical, raising the average trick rate from approximately 69% (vanilla DreamBooth) to 93%.

The study also reveals that spuriousness exists on a spectrum and is highly context-dependent. "Recontextualization" experiments showed that placing a spurious feature in a highly detailed background can sometimes "break" the illusion, suggesting that spurious features often rely on a lack of competing visual information to dominate the model's decision-making process.

Limitations and Future Work

While the method is effective, limitations include occasional diffusion artifacts and potential overfitting to the small reference sets used for fine-tuning. Future work will explore more efficient fine-tuning methods like LoRA (Low-Rank Adaptation) to improve diversity and investigate the security implications of these "robust" spurious features as potential backdoors in AI systems.

Summary updated

Abstract

We propose a method for generating spurious features by leveraging large-scale text-to-image diffusion models. Although the previous work detects spurious features in a large-scale dataset like ImageNet and introduces Spurious ImageNet, we found that not all spurious images are spurious across different classifiers. Although spurious images help measure the reliance of a classifier, filtering many images from the Internet to find more spurious features is time-consuming. To this end, we utilize an existing approach of personalizing large-scale text-to-image diffusion models with available discovered spurious images and propose a new spurious feature similarity loss based on neural features of an adversarially robust model. Precisely, we fine-tune Stable Diffusion with several reference images from Spurious ImageNet with a modified objective incorporating the proposed spurious-feature similarity loss. Experiment results show that our method can generate spurious images that are consistently spurious across different classifiers. Moreover, the generated spurious images are visually similar to reference images from Spurious ImageNet.
BibTeX
@inproceedings{maungmaung2024fine,
  author = {AprilPyone MaungMaung and Huy H. Nguyen and Hitoshi Kiya and Isao Echizen},
  title = {{Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation}},
  booktitle = {ICIP},
  year = {2024},
}
Figure