Diagnosing Shortcut-Induced Rigidity in Continual Learning: The Einstellung Rigidity Index (ERI)

Kai Gu, Weishi ShiarXiv 2025 · DOI · arXiv

Introduction

Continual Learning (CL) research has traditionally focused on mitigating catastrophic forgetting—the loss of previously acquired knowledge when learning new tasks. However, this focus addresses only one side of the stability-plasticity dilemma. This paper investigates the opposite phenomenon: rigidity.

Drawing from cognitive psychology, we identify the Einstellung effect in machine learning. This effect occurs when a model's prior habits or successful strategies prevent it from discovering more optimal solutions for new problems. In CL, this manifests as "shortcut-induced rigidity," where a model over-relies on incidental correlations (shortcuts) from previous tasks, hindering the acquisition of robust, task-specific features.

The Einstellung Rigidity Index (ERI)

To quantify this phenomenon, we introduce the Einstellung Rigidity Index (ERI), a diagnostic suite comprising three metrics:

  1. Adaptation Delay (AD): This measures the relative speed at which a continual learner reaches a target accuracy threshold compared to a baseline model trained from scratch.
  2. Performance Deficit (PD): This quantifies the gap in final accuracy between a scratch-trained model and the continual learner. A positive PD indicates that prior knowledge is actively impeding optimal performance.
  3. Relative Suboptimal Feature Reliance ($SFR_{rel}$): This metric utilizes a masking intervention to determine the extent of a model's dependence on a suspected shortcut. By comparing performance on images with and without the shortcut cue, we can identify if a model is "hooked" on suboptimal features.

Experimental Methodology

We evaluate ERI using a modified CIFAR-100 dataset in a two-phase protocol. In Phase 1, models are trained on eight "clean" superclasses. In Phase 2, they adapt to four new superclasses, two of which contain a deliberate shortcut: a 4x4 magenta patch.

The shortcut injection and masking protocol used to evaluate feature reliance.

This setup allows us to observe whether the model prioritizes the easily learnable magenta patch over the actual semantic features of the new classes.

Key Findings

Our analysis of popular CL methods (e.g., EWC, DER++, GPM, DGR) reveals several critical insights:

  • The "Red Flag" Pattern: High-risk rigidity is characterized by a specific ERI signature: fast initial adaptation (negative AD) coupled with poor final performance (positive PD) and high shortcut reliance (positive $SFR_{rel}$).
  • Anti-Forgetting Backfire: Mechanisms designed to prevent forgetting, such as replay buffers or parameter regularization, can inadvertently "entrench" shortcuts. By protecting early-learned suboptimal features, these methods may prevent the model from ever learning the correct underlying concepts.
  • DGR and Rigidity: Deep Generative Replay (DGR) exhibited the highest levels of rigidity. The generated "memories" appeared to over-regularize the feature space, effectively blocking the model from adapting to new task requirements.
  • Masking as a Diagnostic: For many CL methods, masking the magenta patch during testing actually improved accuracy, confirming that the models were being negatively influenced by the shortcut.

Conclusion and Future Work

The ERI provides a necessary framework for evaluating not just how much a model remembers, but why it performs. While currently tested on synthetic cues and specific architectures like ResNet-18, future work will focus on scaling ERI to more complex datasets and "black-box" shortcut detection using attribution-guided deletion. By diagnosing rigidity, we can move toward CL algorithms that balance stability with the flexibility required for genuine knowledge transfer.

Summary updated

BibTeX
@misc{gu2025diagnosing,
  author = {Kai Gu and Weishi Shi},
  title = {{Diagnosing Shortcut-Induced Rigidity in Continual Learning: The Einstellung Rigidity Index (ERI)}},
  year = {2025},
  eprint = {2510.00475},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.00475},
}
Figure