Robustness Beyond Known Groups with Low-rank Adaptation

Abinitha Gourabathina, Hyewon Jeong, Teya Bergamaschi, Hae Won Park, Collin Stultz

Abstract

Deep learning models trained to optimize average accuracy often exhibit systematic failures on particular subpopulations. In real world settings, the subpopulations most affected by such disparities are frequently unlabeled or unknown, thereby motivating the development of methods that are performant on sensitive subgroups without being pre-specified. However, existing group-robust methods typically assume prior knowledge of relevant subgroups, using group annotations for training or model selection. We propose Low-rank Error Informed Adaptation (LEIA), a simple two-stage method that improves group robustness by identifying a low-dimensional subspace in the representation space where model errors concentrate. LEIA restricts adaptation to this error-informed subspace via a low-rank adjustment to the classifier logits, directly targeting latent failure modes without modifying the backbone or requiring group labels. Using five real-world datasets, we analyze group robustness under three settings: (1) truly no knowledge of subgroup relevance, (2) partial knowledge of subgroup relevance, and (3) full knowledge of subgroup relevance. Across all settings, LEIA consistently improves worst-group performance while remaining fast, parameter-efficient, and robust to hyperparameter choice.

Introduction: The Group Robustness Challenge

Standard machine learning models are usually optimized for average accuracy, which often masks systematic failures on specific subgroups. For example, a facial recognition model might perform well on average but fail significantly on people with darker skin tones. The field of group robustness aims to mitigate these disparities, but most existing solutions require explicit "group labels" (e.g., race, gender) during training. In real-world scenarios, these labels are often expensive, noisy, or entirely unknown. Low-rank Error Informed Adaptation (LEIA) addresses this challenge by improving worst-group accuracy (WGA) without requiring prior knowledge of subgroup identities.

Methodology: Low-rank Error Informed Adaptation

LEIA is a two-stage, parameter-efficient method designed to target latent failure modes by identifying where a model's errors concentrate within its own representation space.

  1. Standard Training (ERM): A base model is trained using standard Empirical Risk Minimization. The feature-extracting "backbone" is then frozen.
  2. Low-rank Adaptation: Using a held-out subset, LEIA identifies where the base model struggles through three steps:
  3. Error Weighting: Assigning higher weights to misclassified or high-loss examples.
  4. Spectral Decomposition: Computing an error-weighted covariance matrix of data representations to find the top-$k$ eigenvectors, defining a low-dimensional "error subspace."
    • Logit Adjustment: Adding a small, learnable linear adjustment to the classifier logits, restricted specifically to the identified subspace.
Figure 1: Overview of LEIA. (A) High-dimensional embedding space. (B) ERM error boundary identification. (C) LEIA's adjusted decision boundary. (D) Cross-section showing how LEIA adjusts probabilities along the error direction to improve worst-group accuracy.

Key Insights and Experimental Results

The core intuition behind LEIA is that model failures are geometrically structured, clustering in specific directions within the representation space. By identifying these directions, the model can be "fixed" without needing to know the explicit name or label of the group being affected.

Performance and Efficiency

The authors evaluated LEIA on five diverse benchmarks: Waterbirds, CelebA, MultiNLI, CivilComments, and CheXpert.

  • State-of-the-Art Robustness: In settings with zero group information, LEIA achieved the best or second-best WGA across all datasets. On CivilComments, it improved WGA by 1.7% over the next best baseline and boosted average accuracy by 6%.
  • Efficiency: LEIA is remarkably lightweight, adding only ~10.8% runtime overhead. For the Waterbirds dataset, it requires learning only 32 parameters in the second stage, compared to millions in competing methods.
  • Stability: The method proved robust to hyperparameter choices, specifically the rank ($k$) and loss-weight sharpness ($\gamma$).

Discussion and Limitations

While LEIA provides a surgical fix for decision boundaries, it relies on the existing feature representations of the frozen backbone. It cannot "learn" new features that the base model missed entirely. Additionally, it assumes that errors concentrate in a low-dimensional subspace. Future work may explore integrating LEIA into the initial training phase or applying it to larger foundation models.

Summary updated

Abstract

Deep learning models trained to optimize average accuracy often exhibit systematic failures on particular subpopulations. In real world settings, the subpopulations most affected by such disparities are frequently unlabeled or unknown, thereby motivating the development of methods that are performant on sensitive subgroups without being pre-specified. However, existing group-robust methods typically assume prior knowledge of relevant subgroups, using group annotations for training or model selection. We propose Low-rank Error Informed Adaptation (LEIA), a simple two-stage method that improves group robustness by identifying a low-dimensional subspace in the representation space where model errors concentrate. LEIA restricts adaptation to this error-informed subspace via a low-rank adjustment to the classifier logits, directly targeting latent failure modes without modifying the backbone or requiring group labels. Using five real-world datasets, we analyze group robustness under three settings: (1) truly no knowledge of subgroup relevance, (2) partial knowledge of subgroup relevance, and (3) full knowledge of subgroup relevance. Across all settings, LEIA consistently improves worst-group performance while remaining fast, parameter-efficient, and robust to hyperparameter choice.
BibTeX
@misc{gourabathina2026robustness,
  author = {Abinitha Gourabathina and Hyewon Jeong and Teya Bergamaschi and Hae Won Park and Collin Stultz},
  title = {{Robustness Beyond Known Groups with Low-rank Adaptation}},
  year = {2026},
  eprint = {2602.06924},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.06924},
}
Figure