Abstract
Deep Neural Networks (DNNs) often rely on "shortcut learning," where they associate spurious features—such as backgrounds or co-occurring objects—with specific classes rather than learning the core object characteristics. This paper introduces a method to fine-tune text-to-image diffusion models to generate an unlimited supply of these spurious images. By incorporating a novel Spurious Feature Similarity Loss (SFSL), the researchers demonstrate that synthetic images can be crafted to "universally" trick various model architectures, including ResNets and Vision Transformers. This approach provides a robust framework for auditing and debugging computer vision systems against real-world failures.
Introduction
Deep learning models are notoriously prone to learning spurious correlations. For instance, a model might identify a "hummingbird" based on the presence of red tubular flowers rather than the bird itself.
While datasets like Spurious ImageNet provide examples of these failures, they suffer from data scarcity and a lack of universality—features that trick one model may not trick another. This research addresses these limitations by using generative AI to create a "synthetic laboratory" for spurious feature analysis.
Methodology
The authors leverage DreamBooth to fine-tune Stable Diffusion on a small set of reference spurious images. To ensure the generated images prioritize the misleading "shortcuts" over general visual fidelity, they introduce the Spurious Feature Similarity Loss (SFSL).
This loss function utilizes an adversarially robust model as a feature extractor, calculating the cosine similarity between the neural representations of generated and reference images. This process forces the diffusion model to capture the specific textures and shapes (e.g., flower patterns or flagpole structures) that characterize the spurious class.
Key Results and Insights
The generated images achieved significantly higher "spurious accuracy"—the rate at which classifiers are successfully misled—compared to the original dataset. For example, in the "Gondola" class, the synthetic images reached 100% spurious accuracy across four different architectures (ResNet-50 v1/v2, Robust ResNet, and ViT). Ablation studies confirmed that the SFSL was critical, raising the average trick rate from approximately 69% (vanilla DreamBooth) to 93%.
The study also reveals that spuriousness exists on a spectrum and is highly context-dependent. "Recontextualization" experiments showed that placing a spurious feature in a highly detailed background can sometimes "break" the illusion, suggesting that spurious features often rely on a lack of competing visual information to dominate the model's decision-making process.
Limitations and Future Work
While the method is effective, limitations include occasional diffusion artifacts and potential overfitting to the small reference sets used for fine-tuning. Future work will explore more efficient fine-tuning methods like LoRA (Low-Rank Adaptation) to improve diversity and investigate the security implications of these "robust" spurious features as potential backdoors in AI systems.
Summary updated