Invariance Pair-Guided Learning: Enhancing Robustness in Neural Networks

Martin Surner, Abdelmajid Khelil, Ludwig Bothmann

Abstract

Out-of-distribution generalization of machine learning models remains challenging since the models are inherently bound to the training data distribution. This especially manifests when the learned models rely on spurious correlations. Most existing approaches apply data manipulation, representation learning, or learning strategies to achieve generalizable models. Unfortunately, these approaches usually require multiple training domains, group labels, specialized augmentation, or pre-processing. We propose a novel approach that addresses these limitations by providing a technique to guide the neural network through the training phase. We first establish input pairs, representing the spurious attribute and describing the invariance, a characteristic that should not affect the outcome of the model. Based on these pairs, we form a corrective gradient complementing the traditional gradient descent approach. We further make this correction mechanism adaptive based on a predefined invariance condition. Experiments on ColoredMNIST, Waterbird-100, and CelebA datasets demonstrate the effectiveness of our approach and the robustness to group shifts.

Introduction

Deep learning models often exhibit a tendency to exploit "shortcuts" or spurious correlations within training data. For instance, if a dataset of waterbirds predominantly features aquatic backgrounds, a model may learn to identify the background rather than the avian subject. Consequently, these models fail when presented with out-of-distribution (OOD) data, such as a waterbird on dry land.

Current methods to mitigate this issue often necessitate extensive metadata, multiple training environments, or complex data augmentation. This paper introduces Invariance Pair-Guided Learning (IPG), a surgical training technique designed to guide a model's internal representations toward invariant features, ensuring it focuses on the correct semantic concepts rather than misleading noise.

The IPG Framework

The IPG framework operates by explicitly teaching the network what to ignore. It consists of three primary pillars:

1. Invariance Pairs

The foundation of the method is the use of invariance pairs ($\mathcal{I}$). These are pairs of samples that share the same class label but differ significantly in a spurious attribute. By presenting these pairs to the model, the training process can identify and penalize reliance on non-essential features.

Examples of invariance pairs for (b) ColoredMNIST, (c) Waterbird-100, and (d) CelebA datasets.

2. The Rationale Matrix

To quantify the model's internal logic, the authors introduce a "rationale matrix" ($R$). This matrix maps the model's internal feature representations to its final classification weights. By comparing the rationales generated for an invariance pair, the framework can measure the extent to which the model is influenced by spurious features (e.g., the color of a digit or the background of a bird).

3. Corrective Gradient and Adaptive Scaling

IPG modifies the standard training loop with a two-step gradient update:

  • Correction Step: A corrective gradient ($g_d$) is calculated to minimize the discrepancy between the rationales of the invariance pairs. This pushes the model toward a representation that is invariant to the spurious attribute.
  • Adaptive Scaling: An invariance condition based on KL divergence is applied. If the model's predictions for the two paired images differ significantly, the standard loss gradient is rescaled (effectively slowed down). This forces the model to prioritize learning the invariance before proceeding with standard classification optimization.

For scenarios where pairs are not readily available, the authors propose IPG-AA, which utilizes adversarial augmentation to generate these pairs automatically during the training process.

Experimental Evaluation

The researchers evaluated IPG against several state-of-the-art robustness methods across three benchmark datasets:

  • ColoredMNIST: A dataset where digit color is a strong but misleading predictor of the label. IPG achieved 72.7% test accuracy, nearing the theoretical maximum of 75%.
  • Waterbird-100: A dataset where bird species are correlated with specific backgrounds. IPG demonstrated a significant breakthrough in worst-group accuracy, reaching 68.62%. This represents an 11.9 percentage point improvement over the leading specialized "explanation-guided" methods.
  • CelebA: A real-world dataset where gender is spuriously correlated with hair color. IPG remained competitive, though slightly behind methods like JTT, likely due to the complexity of generating high-quality invariance pairs for human faces.

Discussion and Insights

The success of IPG stems from its "guiding" nature. Unlike standard regularization, which simply adds a penalty term, IPG actively steers the weights toward invariant representations. The adaptive scaling mechanism is particularly crucial; it prevents the model from "cheating" by using spurious features to quickly reduce training loss.

Visual analysis using t-SNE plots confirmed these findings. While standard Empirical Risk Minimization (ERM) models clustered data based on spurious features (like color), IPG-trained models successfully clustered data by their actual semantic class, effectively ignoring the "noise."

Limitations and Future Work

While effective, IPG has some constraints:

  1. Pair Quality: The performance on complex datasets like CelebA is sensitive to the quality of the invariance pairs.
  2. Computational Cost: The additional inference steps for pairs increase training time.
  3. Task Scope: Currently limited to classification, with future work needed for regression tasks.
  4. Multiple Shortcuts: Future iterations will explore handling datasets with multiple simultaneous spurious correlations.

In conclusion, IPG offers a robust and targeted approach to training neural networks that are resilient to distribution shifts, paving the way for more reliable machine learning systems in real-world applications.

Summary updated

Abstract

Out-of-distribution generalization of machine learning models remains challenging since the models are inherently bound to the training data distribution. This especially manifests, when the learned models rely on spurious correlations. Most of the existing approaches apply data manipulation, representation learning, or learning strategies to achieve generalizable models. Unfortunately, these approaches usually require multiple training domains, group labels, specialized augmentation, or pre-processing to reach generalizable models. We propose a novel approach that addresses these limitations by providing a technique to guide the neural network through the training phase. We first establish input pairs, representing the spurious attribute and describing the invariance, a characteristic that should not affect the outcome of the model. Based on these pairs, we form a corrective gradient complementing the traditional gradient descent approach. We further make this correction mechanism adaptive based on a predefined invariance condition. Experiments on ColoredMNIST, Waterbird-100, and CelebA datasets demonstrate the effectiveness of our approach and the robustness to group shifts.
BibTeX
@misc{surner2025invariance,
  author = {Martin Surner and Abdelmajid Khelil and Ludwig Bothmann},
  title = {{Invariance Pair-Guided Learning: Enhancing Robustness in Neural Networks}},
  year = {2025},
  eprint = {2502.18975},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.18975},
}

Loading related content...

Figure