Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers

Wanqian Yang, Polina Kirichenko, Micah Goldblum, Andrew Gordon Wilson

Deep neural networks are notoriously susceptible to "shortcut learning"—a phenomenon where models rely on spurious correlations (like image backgrounds or watermarks) rather than essential semantic structures to make predictions. While it was previously theorized that generative models might naturally avoid this by learning comprehensive data representations, recent findings show they are just as vulnerable.

To address this, researchers have introduced Chroma-VAE, a novel framework that actively prevents neural networks from relying on these shortcuts. By intelligently partitioning a Variational Auto-Encoder’s (VAE) latent space, Chroma-VAE "traps" the shortcut in a small subspace, leaving the remaining representation clean, robust, and ready for reliable classification.

The Problem: The "Laziness" of Deep Learning

Deep learning models are fundamentally optimization machines; if a dataset contains a simple, easy-to-learn feature that correlates with a label (a "shortcut"), the model will exploit it. For example, a model trained to identify cows might simply learn to detect green pastures. If presented with a cow on a beach, the model fails.

This vulnerability poses significant safety risks in real-world applications. In medical imaging, models have been known to predict pneumonia not by analyzing lung tissue, but by detecting hospital-specific metal tokens or text artifacts on X-ray scans. The challenge lies in forcing a model to ignore these "cheap" shortcuts and focus on the harder, genuinely predictive semantic features.

The Solution: Chroma-VAE

Inspired by chromatography—the chemical process of separating a mixture into its individual components—the authors propose Chroma-VAE. The method leverages the fact that shortcuts are typically low-information but highly predictive.

Chroma-VAE architecture demonstrating latent space partitioning.

The Chroma-VAE framework operates in two main phases:

  1. Latent Space Partitioning (The "Trap"): The VAE's latent space is split into two disjoint parts: $z_1$ and $z_2$. A classifier is attached only to the $z_1$ subspace. Because the network's inductive bias naturally pushes the simplest predictive information (the shortcut) into $z_1$ to minimize the classifier's loss, the shortcut becomes isolated. Meanwhile, the VAE's overall reconstruction loss forces the combined latent space $(z_1, z_2)$ to represent the entire image.
  2. The Clean Slate: With the shortcut effectively sequestered in $z_1$, the $z_2$ subspace is left with a "shortcut-invariant" representation—capturing the core semantic features without the spurious correlation. A secondary, robust classifier is then trained exclusively on this clean $z_2$ representation.

By sampling only from $z_1$, partial reconstructions reveal images containing only the shortcut (e.g., just a color patch). Conversely, sampling from $z_2$ reconstructs the actual object (like a face or digit) entirely free of the shortcut.

Experimental Validation

Chroma-VAE was evaluated across several benchmarks specifically designed to trigger shortcut learning:

  • ColoredMNIST: A dataset where digit color spuriously correlates with the label during training but is reversed during testing.
  • CelebA: Real-world face images evaluated for tasks like predicting "Blond" (where Gender acts as a shortcut) and "Attractive" (where Smiling is a shortcut).
  • Chest X-ray: A medical dataset where hospital-specific background artifacts serve as dangerous shortcuts for pneumonia detection.

Key Results

Chroma-VAE significantly outperformed standard Empirical Risk Minimization (ERM) and other robust training methods, particularly when shortcuts were perfectly correlated with labels during training.

  • ColoredMNIST: On adversarial test sets where the shortcut fails, standard models plummeted to ~11-17% accuracy. Chroma-VAE maintained a robust 72.4% accuracy.
  • CelebA (Blond/Gender): Chroma-VAE achieved a worst-group accuracy of 54.4%, nearly doubling the performance of a standard Naive-Class model (29.4%).
  • Chest X-ray: For pneumonia detection, Chroma-VAE achieved the highest worst-group accuracy (57.8%). In stark contrast, the standard model's accuracy collapsed to 10.3% when the hospital-specific shortcut was removed.

Insights and Future Directions

A critical insight from this research is that generative models are not a silver bullet against shortcut learning. From an information-theoretic perspective, shortcuts are highly efficient data compressions. Under limited capacity, models will inherently prefer them over complex semantic features unless explicitly architected otherwise.

While highly effective, Chroma-VAE does have limitations. Because it relies on VAEs, it can struggle to model highly complex, high-resolution natural scenes compared to state-of-the-art Diffusion models or GANs. Furthermore, the method is most effective when the shortcut is "simpler" than the semantic feature; highly complex shortcuts may be harder to isolate in a small subspace.

Future work will likely explore extending this partitioning logic to more powerful generative frameworks, such as Diffusion models, to handle increasingly complex real-world datasets. There is also potential in introducing priors that explicitly encourage different latent subspaces to correspond to specific, interpretable features, paving the way for more transparent and robust AI systems.

Summary updated

Abstract

Deep neural networks are susceptible to shortcut learning, using simple features to achieve low training loss without discovering essential semantic structure. Contrary to prior belief, we show that generative models alone are not sufficient to prevent shortcut learning, despite an incentive to recover a more comprehensive representation of the data than discriminative approaches. However, we observe that shortcuts are preferentially encoded with minimal information, a fact that generative models can exploit to mitigate shortcut learning. In particular, we propose Chroma-VAE, a two-pronged approach where a VAE classifier is initially trained to isolate the shortcut in a small latent subspace, allowing a secondary classifier to be trained on the complementary, shortcut-free latent subspace. In addition to demonstrating the efficacy of Chroma-VAE on benchmark and real-world shortcut learning tasks, our work highlights the potential for manipulating the latent space of generative classifiers to isolate or interpret specific correlations.
BibTeX
@inproceedings{yang2022chroma,
  author = {Wanqian Yang and Polina Kirichenko and Micah Goldblum and Andrew Gordon Wilson},
  title = {{Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=WWVcsfI0jGH},
}
Figure