Regulating Internal Alignment Flows for Robust Learning Under Spurious Correlations

Rajeev Ranjan Dwivedi, Mohammedkaif Mohammedrafiq Kalagond, Niramay M.Patel, Vinod K. KurmiICLR 2026 · OpenReview

Deep learning models are notoriously "lazy." If a dataset contains a shortcut—like identifying a "waterbird" simply because there is water in the background—the model will exploit it. This reliance on spurious correlations leads to high average accuracy but catastrophic failures on minority groups (e.g., a waterbird on land).

In a new paper, researchers introduce Alignment-Gated Suppression (AGS), a lightweight, "plug-and-play" regularizer designed to stop models from obsessing over these spurious correlations—all without needing expensive group labels.

The Challenge of Spurious Correlations

Standard training via Empirical Risk Minimization (ERM) encourages models to minimize error by any means necessary. Often, the easiest path is to latch onto "easy" features like backgrounds, gender stereotypes, or lighting that correlate with labels in the training set but fail in the real world.

Existing solutions usually fall into two camps:

  1. Group-aware methods: These require "group labels" (e.g., knowing exactly which images have "water" backgrounds), which are often missing or expensive to produce.
  2. Data or Loss interventions: These modify the input data or the loss function but don't directly control the internal pathways of the network where shortcut learning actually happens.

AGS fills this gap by intervening inside the model during training, targeting the specific neurons that propagate "overconfident" shortcut energy.

How Alignment-Gated Suppression Works

AGS is a group-agnostic regularizer that identifies and suppresses "shortcut" neurons by tracking their class-conditional alignment energy. The mechanism operates through four key steps:

  1. Alignment Energy: For every neuron, the model tracks an "alignment score." This score measures how much a specific neuron contributes to a class-consistent prediction, weighted by the model's own confidence. A more negative score indicates stronger, more confident support for a class.
  2. EMA Smoothing: To prevent the model from reacting to noisy individual batches, AGS maintains an Exponential Moving Average (EMA) of these scores over time.
  3. Percentile-Gated Suppression: At each training step, the model identifies the "most extreme" contributors—the neurons in the bottom $q$-th percentile of alignment energy. These are the neurons most likely to be driving overconfident shortcuts.
  4. Multiplicative Decay: AGS applies a multiplicative decay (shrinkage) to the weights of these flagged neurons. This effectively "turns down the volume" on shortcut pathways, forcing the model to rely on more robust, diverse features.

Crucially, AGS is scale-invariant thanks to its percentile-based gating and requires no architectural changes.

Datasets and Benchmarks

The authors evaluated AGS on several rigorous benchmarks for spurious correlations, including Waterbirds (background bias), CelebA (gender/hair color bias), and BAR (action/environment bias). They also introduced a skewed COCO Gender/Object Bias benchmark to test for stereotypical bias (e.g., pairing "male" with "sports" and "female" with "kitchen").

COCO Gender/Object Bias Results

Key Insights

  • Confidence is a Double-Edged Sword: Shortcuts often lead to high-confidence predictions. By weighting the alignment energy by the softmax probability, AGS specifically targets the features the model is "too sure" about.
  • Internal Locus of Control: Unlike reweighting data, AGS acts on the internal conduits of the network, providing a more direct way to reshape learning dynamics.
  • No Labels Needed: The model can self-identify and regulate its own shortcut pathways simply by looking at the distribution of alignment energy, removing the need to know what the spurious feature is beforehand.

Key Results

AGS consistently outperforms or matches state-of-the-art methods across major benchmarks while adding less than 5% computational overhead.

Results on BAR, CelebA, and Waterbirds
  • CelebA: AGS achieved a 93.95% worst-group (conflicting) accuracy, a massive jump over the previous state-of-the-art (EvA-E at 88.74%) and standard ERM (52.52%).
  • BAR: It delivered the strongest average accuracy at 76.09%, a gain of +15.58 points over standard ERM.
  • Calibration: Beyond accuracy, AGS significantly improved model calibration, ensuring the model's confidence levels more accurately reflect its actual likelihood of being correct.

Limitations and Future Work

While powerful, AGS has a few limitations. Because alignment is estimated on-batch, very small or highly imbalanced batches can lead to noisy estimates and "over-suppression." It also requires a brief "warm-up" period (e.g., 5 epochs) to let the EMA buffers stabilize before suppression begins. Finally, the current implementation primarily targets the final classifier layer; extending this to intermediate layers (like attention heads in Transformers) or non-vision tasks remains an exciting avenue for future work.

Summary updated

Abstract

Deep models often exploit spurious correlations (e.g., backgrounds or dataset artifacts), hurting worst-group performance. We propose \textbf{Alignment-Gated Suppression (AGS)}, a lightweight, plug-in regularizer that intervenes inside the network during training. AGS tracks a class-conditional, confidence-weighted contribution for each neuron (more negative $\Leftrightarrow$ stronger support) and applies a percentile-based, multiplicative decay to the most extreme contributors, reducing overconfident shortcut pathways while leaving other features relatively more influential. AGS integrates with standard ERM, requires no group labels, and adds $<5\%$ training overhead. We provide analysis linking AGS to minority-margin gains, path-norm-like capacity control, and stability benefits via EMA-smoothed gating. Empirically, AGS improves worst-group accuracy and calibration vs.\ ERM and is competitive with state-of-the-art methods across spurious-correlation benchmarks (e.g., Waterbirds, CelebA, BAR, COCO), while maintaining strong average accuracy. These results suggest that regulating internal alignment flow is a simple and scalable route to robustness without group labels.
BibTeX
@inproceedings{dwivedi2026regulating,
  author = {Rajeev Ranjan Dwivedi and Mohammedkaif Mohammedrafiq Kalagond and Niramay M.Patel and Vinod K. Kurmi},
  title = {{Regulating Internal Alignment Flows for Robust Learning Under Spurious Correlations}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=L2L1hi0FGj},
}
Figure