MAP: Towards Balanced Generalization of IID and OOD through Model-Agnostic Adapters

Min Zhang, Junkun Yuan, Yue He, Wenbin Li, Zhengyu Chen, Kun Kuang

Introduction

In machine learning, models are typically trained under the assumption that training and testing data are Independent and Identically Distributed (IID). However, real-world deployments often face Out-of-Distribution (OOD) data, where environmental shifts—such as different lighting or equipment—can degrade performance. While existing OOD generalization methods like Invariant Risk Minimization (IRM) or Variance Risk Extrapolation (VREx) focus on finding "invariant features" to handle these shifts, they often do so at a significant cost to IID performance. This "IID-OOD dilemma" creates a risky trade-off for practitioners who need models that perform well regardless of whether a shift occurs.

This paper introduces MAP (Model-Agnostic Adapters), a framework designed to achieve balanced generalization. MAP allows models to maintain high accuracy on standard IID data while remaining robust to distribution shifts.

The MAP Framework

The core innovation of MAP is treating IID and OOD learning as two distinct but coupled tasks. Instead of forcing a single model to satisfy both objectives simultaneously, MAP introduces Auxiliary Adapter Layers (AALs). These are lightweight modules inserted into a standard OOD model to capture the inductive bias necessary for IID performance.

The MAP framework utilizes auxiliary adapter layers and bilevel optimization to balance IID and OOD objectives.

To manage the interaction between the main model and these adapters, the authors employ a Bilevel Optimization (BLO) strategy:

  • Inner Level: The main OOD model is optimized using an OOD loss (e.g., VREx) to learn invariant features from training data.
  • Outer Level: The adapters are updated using an IID loss (Empirical Risk Minimization) on a validation set. This feedback loop ensures the adapters learn features that the OOD-focused main model might otherwise ignore.

To ensure efficiency, the authors derived a first-order approximation for this bilevel process, making MAP's training time comparable to standard OOD methods.

Key Insights: Visualizing the Dilemma

By analyzing BatchNorm statistics (running means and variances), the researchers demonstrated that IID and OOD methods prioritize different features. IID methods tend to focus on "easy-to-learn" variant features, whereas OOD methods target "hard-to-learn" invariant features. MAP bridges this gap by allowing the model to leverage both.

A critical metric introduced to evaluate this balance is the Harmonic Mean (HM) of IID and OOD accuracy. A high HM indicates that a model generalizes well across both scenarios without sacrificing one for the other.

Experimental Results

MAP was evaluated across six diverse datasets, including synthetic shifts (ColoredMNIST, ColoredCOCO) and real-world shifts (NICO, WILDSCamelyon).

Comparison of OOD accuracy, IID accuracy, and Harmonic Mean (HM) across various methods, highlighting MAP's balanced performance.

Key findings include:

  • Superior Balance: MAP consistently achieved the highest Harmonic Mean across all datasets. On ColoredMNIST, adding MAP to the VREx method improved the HM from 22.9% to 60.6%.
  • IID Recovery: While standard OOD methods often drop IID accuracy by 30-50% compared to a simple baseline (ERM), MAP recovers most of this performance while maintaining OOD robustness.
  • Synergistic Effects: In some cases, MAP even improved OOD performance, suggesting that learning IID features can provide a more stable foundation for invariant feature discovery.

Conclusion

MAP offers a "plug-and-play" solution for existing OOD algorithms, effectively resolving the IID-OOD dilemma. By decoupling the learning objectives through adapters and bilevel optimization, it provides a robust framework for real-world machine learning deployment. Future work may explore its application in adversarial robustness and more complex tasks like object detection or segmentation.

Summary updated

Abstract

Deep learning has achieved tremendous success in recent years, but most of these successes are built on an independent and identically distributed (IID) assumption. This somewhat hinders the application of deep learning to the more challenging out-of-distribution (OOD) scenarios. Although many OOD methods have been proposed to address this problem and have obtained good performance on testing data that is of major shifts with training distributions, interestingly, we experimentally find that these methods achieve excellent OOD performance by making a great sacrifice of the IID performance. We call this finding the IID-OOD dilemma. Clearly, in real-world applications, distribution shifts between training and testing data are often uncertain, where shifts could be minor, and even close to the IID scenario, and thus it is truly important to design a deep model with the balanced generalization ability between IID and OOD. To this end, in this paper, we investigate an intriguing problem of balancing IID and OOD generalizations and propose a novel Model Agnostic adaPters (MAP) method, which is more reliable and effective for distribution-shift-agnostic real-world data. Our key technical contribution is to use auxiliary adapter layers to incorporate the inductive bias of IID into OOD methods. To achieve this goal, we apply a bilevel optimization to explicitly model and optimize the coupling relationship between the OOD model and auxiliary adapter layers. We also theoretically give a first-order approximation to save computational time. Experimental results on six datasets successfully demonstrate that MAP can greatly improve the performance of IID while achieving good OOD performance.
BibTeX
@inproceedings{zhang2023map,
  author = {Min Zhang and Junkun Yuan and Yue He and Wenbin Li and Zhengyu Chen and Kun Kuang},
  title = {{MAP: Towards Balanced Generalization of IID and OOD through Model-Agnostic Adapters}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01095},
}
Figure