Abstract
Neural networks are known to use spurious correlations such as background information for classification. While prior work has looked at spurious correlations that are widespread in the training data, in this work, we investigate how sensitive neural networks are to rare spurious correlations, which may be harder to detect and correct, and may lead to privacy leaks. We introduce spurious patterns correlated with a fixed class to a few training examples and find that it takes only a handful of such examples for the network to learn the correlation. Furthermore, these rare spurious correlations also impact accuracy and privacy. We empirically and theoretically analyze different factors involved in rare spurious correlations and propose mitigation methods accordingly. Specifically, we observe that $\ell_2$ regularization and adding Gaussian noise to inputs can reduce the undesirable effects.
Introduction
Machine learning models are notorious for picking up on "spurious correlations"—shortcuts where the model relies on a feature (like a snowy background) to identify an object (like a wolf) because that feature was common in the training data. While researchers have extensively studied widespread spurious correlations, this paper investigates the "long tail": rare spurious correlations.
Neural networks are incredibly sensitive to these rare signals, often learning to associate a specific class with a background pattern after seeing it in as few as 1 to 3 training examples. This "shortcut learning" doesn't just hurt generalization; it creates significant privacy risks. If a model associates a specific person with a unique background because of a single training image, an adversary can present that background to the model and infer the individual's presence in the training set—a classic membership inference leak.
Experimental Framework
The researchers used a two-pronged approach involving empirical studies and theoretical modeling. They artificially injected spurious patterns into standard benchmarks like MNIST, Fashion-MNIST, and CIFAR-10. These patterns were categorized into two main types:
- Localized (S1-S3): Small, fixed-color squares placed in specific locations within the image.
- Dispersed (R1-R3): Random noise patterns spread across the entire image.
To measure the impact, the authors defined a "Spurious Score"—the increase in the model's confidence for a target class when the spurious pattern is added to a clean test image. They also validated their findings using the NICO++ dataset, which contains natural spurious correlations (e.g., identifying an "airplane" based on an "autumn" context).
Key Insights and Results
The "Handful" Rule and Phase Transitions
In datasets containing 60,000 images, it takes only 1 to 3 examples for a neural network to develop a significant spurious correlation. This learning is not gradual; theoretical analysis of a linear "concatenate model" and empirical data both reveal a "sharp phase transition." Once a tiny threshold of evidence is met, the model's reliance on the shortcut jumps from near-zero to significant levels.
Privacy as the Hidden Victim
Rare spurious correlations act like "fingerprints." The authors found that images containing these rare patterns are much more vulnerable to Membership Inference Attacks (MIA) than standard training images. Because the model becomes hyper-sensitive to these specific features, their presence in a query image makes it much easier for an attacker to identify that the image was part of the training set.
Pattern Complexity
Interestingly, dispersed patterns (like random noise) are often learned more easily by the network than small, concentrated localized squares. This suggests that the total number of pixels affected by a correlation may be more important than the intensity or localization of the signal.
Mitigation and Limitations
The study tested several common regularization techniques to "unlearn" these shortcuts:
- Effective Methods: $\ell_2$ regularization (weight decay) and adding Gaussian noise to inputs effectively reduced spurious scores and improved robustness, though often at a slight cost to clean accuracy.
- Ineffective Methods: Gradient clipping—a staple of differential privacy—was largely ineffective at stopping the model from learning these rare shortcuts. Furthermore, data deletion methods like influence functions failed to effectively remove the correlation once it was baked into the weights.
While the study primarily focuses on artificial patterns in image classification, the natural validation on NICO++ suggests these findings are broadly applicable. Future work is needed to design architectures that are inherently "shortcut-resistant" from the start, as these correlations are incredibly "sticky" once learned.
Summary updated