Annotation-Free Group Robustness via Loss-Based Resampling

Mahdi Ghaznavi, Hesam Asadollahzadeh, HamidReza Yaghoubi Araghi, Fahimeh Hosseini Noohdani, Mohammad Hossein Rohban, Mahdieh Soleymani BaghshaharXiv 2023 · DOI · arXiv

TL;DR

This paper introduces Loss-Based Feature Reweighting (LFR), a novel method to improve the robustness of image classification models against spurious correlations without requiring explicit group annotations. LFR achieves state-of-the-art performance by intelligently selecting samples based on their loss values from an ERM-trained model to retrain the last layer, effectively mitigating reliance on non-causal features.

What Problem Does This Paper Address?

Deep neural networks, while powerful, often learn to rely on spurious correlations in training data rather than true causal features. This leads to a significant drop in performance when the data distribution shifts, particularly affecting minority groups that don't exhibit the dominant spurious pattern. Existing methods to combat this often require costly group annotations or extensive retraining of the entire model. This paper addresses the challenge of achieving group robustness efficiently and without the need for explicit group labels, which are frequently unavailable in real-world scenarios.

Methods

The core of the proposed method, Loss-Based Feature Reweighting (LFR), involves a two-stage process:

  1. Initial Training: A standard neural network is trained using Empirical Risk Minimization (ERM) on a class-balanced dataset. This initial model, while potentially biased by spurious correlations, learns useful feature representations.
  2. Loss-Based Resampling and Retraining:
    • The ERM-trained model is evaluated on a small, held-out portion of the training data.
  3. Samples are categorized into correctly classified (low loss) and misclassified (high loss) groups for each class.
  4. Crucially, an equal number of samples are selected from each class: high-loss samples from misclassified instances and low-loss samples from correctly classified instances. The intuition is that high-loss misclassified samples are likely from minority groups lacking the spurious attribute, while low-loss correctly classified samples are likely from majority groups that do exhibit the spurious attribute.
  5. The last layer of the network is then reinitialized and retrained on this balanced subset of data.

This approach leverages the loss values as a proxy for identifying samples belonging to minority or majority groups without explicit labels, and then uses this information to create a balanced dataset for robust retraining of the final classification layer.

Datasets

The paper evaluates LFR on two popular datasets:

  • Waterbirds: This dataset features images of waterbirds and landbirds, with a spurious correlation between the bird type and the background (water vs. land).
  • CelebA: This dataset is used for hair color classification, where a spurious correlation exists between hair color (blonde) and gender (women).

The authors also created six versions of these datasets with varying degrees of spurious correlation (from 70% to 100%) to provide a more comprehensive evaluation.

Key Insights and Ideas

  • Loss as a Proxy for Group Membership: The paper demonstrates that loss values from an ERM-trained model can effectively act as a signal for identifying samples belonging to minority or majority groups, even without explicit annotations. High loss in misclassified samples suggests a lack of spurious correlation (minority), while low loss in correctly classified samples suggests the presence of spurious correlation (majority).
  • Retraining the Last Layer is Sufficient: Similar to prior work, LFR shows that retraining only the last layer of a pre-trained network is an efficient way to achieve robustness without needing to fine-tune the entire model.
  • Importance of Balanced Sampling: The success of LFR hinges on selecting an equal number of samples from each class, specifically balancing high-loss misclassified and low-loss correctly classified instances. This ensures that minority groups are adequately represented during retraining.
  • Visualizing Attention Shifts: Grad-CAM visualizations reveal that LFR successfully shifts the model's attention from spurious features (like background) to core attributes, improving robustness for both majority and minority group examples.

Key Results

  • State-of-the-Art Performance: LFR consistently outperforms previous methods, including DFR (which requires group annotations) and AFR (another annotation-free method), especially in settings with high spurious correlation.
  • Efficiency: LFR is described as "extremely fast and straightforward," requiring only a small subset of data for last-layer retraining.
  • Robustness Across Correlation Levels: While LFR excels at high spurious correlations, it also performs well across a wide range of correlation levels.
  • Ablation Study: An ablation study comparing LFR with a simplified version called Classification-based Feature Re-weighting (CFR), which uses random selection instead of loss-based selection, highlights the critical role of loss-based sampling.

Limitations and Future Work

  • Hyperparameter Tuning: The paper mentions that the optimal sample size for retraining is found via hyperparameter selection based on worst group accuracy on a validation set.
  • Low Spurious Correlation Nuance: The authors note that at very low spurious correlations (e.g., 70%), the impact of loss-based selection diminishes, and random selection of misclassified samples can sometimes yield better results. This suggests that in such cases, misclassification might be due to factors other than spurious features.
  • Future Work: The paper implicitly suggests exploring more sophisticated methods for sample selection or group inference in low-correlation scenarios. The effectiveness of LFR across various datasets and its ability to improve attention to core attributes also open avenues for further research in robust AI.

Summary updated

Abstract

It is well-known that training neural networks for image classification with empirical risk minimization (ERM) makes them vulnerable to relying on spurious attributes instead of causal ones for prediction. Previously, deep feature re-weighting (DFR) has proposed retraining the last layer of a pre-trained network on balanced data concerning spurious attributes, making it robust to spurious correlation. However, spurious attribute annotations are not always available. In order to provide group robustness without such annotations, we propose a new method, called loss-based feature re-weighting (LFR), in which we infer a grouping of the data by evaluating an ERM-pre-trained model on a small left-out split of the training data. Then, a balanced number of samples is chosen by selecting high-loss samples from misclassified data points and low-loss samples from correctly-classified ones. Finally, we retrain the last layer on the selected balanced groups to make the model robust to spurious correlation. For a complete assessment, we evaluate LFR on various versions of Waterbirds and CelebA datasets with different spurious correlations, which is a novel technique for observing the model's performance in a wide range of spuriosity rates. While LFR is extremely fast and straightforward, it outperforms the previous methods that do not assume group label availability, as well as the DFR with group annotations provided, in cases of high spurious correlation in the training data.
BibTeX
@misc{ghaznavi2023annotation,
  author = {Mahdi Ghaznavi and Hesam Asadollahzadeh and HamidReza Yaghoubi Araghi and Fahimeh Hosseini Noohdani and Mohammad Hossein Rohban and Mahdieh Soleymani Baghshah},
  title = {{Annotation-Free Group Robustness via Loss-Based Resampling}},
  year = {2023},
  eprint = {2312.04893},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.04893},
}

Loading related content...

Figure