Abstract
Deep neural networks are susceptible to shortcut learning, using simple features to achieve low training loss without discovering essential semantic structure. Contrary to prior belief, we show that generative models alone are not sufficient to prevent shortcut learning, despite an incentive to recover a more comprehensive representation of the data than discriminative approaches. However, we observe that shortcuts are preferentially encoded with minimal information, a fact that generative models can exploit to mitigate shortcut learning. In particular, we propose Chroma-VAE, a two-pronged approach where a VAE classifier is initially trained to isolate the shortcut in a small latent subspace, allowing a secondary classifier to be trained on the complementary, shortcut-free latent subspace. In addition to demonstrating the efficacy of Chroma-VAE on benchmark and real-world shortcut learning tasks, our work highlights the potential for manipulating the latent space of generative classifiers to isolate or interpret specific correlations.
Introduction
Machine learning models are notoriously prone to "shortcut learning"—relying on easy but spurious correlations rather than the underlying semantic logic. For instance, a model might identify "cows" by the presence of green grass rather than the animal's shape. While effective on training data, these shortcuts lead to catastrophic failure in out-of-distribution scenarios, such as a cow on a beach. This issue is particularly critical in high-stakes domains like medical imaging, where models might latch onto hospital-specific artifacts instead of pathology.
Methodology: The Chroma-VAE Framework
Chroma-VAE addresses this by partitioning the latent space of a Variational Auto-Encoder (VAE) to "trap" shortcuts. The process involves three key stages:
- Shortcut Isolation ($z_1$): The VAE's latent space is split into two subspaces, $z_1$ and $z_2$. A classifier is trained exclusively on $z_1$. Due to the simplicity bias of neural networks, the model naturally encodes the most predictive, low-information shortcuts into this limited subspace.
- Semantic Reservoir ($z_2$): Simultaneously, the VAE must reconstruct the input image using both $z_1$ and $z_2$. Since the shortcut is sequestered in $z_1$, the $z_2$ subspace is forced to capture the remaining complex semantic features necessary for accurate reconstruction.
- Robust Classification: A final classifier is trained solely on the $z_2$ subspace, which is now "shortcut-invariant."
Experimental Evaluation
The authors evaluated Chroma-VAE across several benchmarks, including ColoredMNIST, CelebA, and real-world Chest X-ray datasets. A major advantage of this approach is "visual debugging." By generating partial reconstructions from $z_2$, researchers can verify that the shortcut has been successfully removed.
Key Results
- ColoredMNIST: In adversarial tests where shortcut colors were reversed, Chroma-VAE maintained ~72% accuracy, whereas standard models dropped to ~18%.
- Chest X-rays: In pneumonia detection, Chroma-VAE achieved a 57.8% worst-group accuracy, significantly outperforming the 10.3% achieved by naive models.
- Independence from Labels: Crucially, Chroma-VAE achieved these results without needing to know which images contained shortcuts during training, making it highly practical for real-world use.
Conclusion
Chroma-VAE demonstrates that generative models can be powerful tools for mitigating shortcut learning when their latent spaces are explicitly structured. By exploiting the information-light nature of shortcuts, the framework provides a robust, interpretable method for training models that generalize better to real-world data without requiring expensive group labels. Future work may explore integrating more advanced generative architectures, such as diffusion models, to handle higher-complexity data.
Summary updated