Robust Generalization despite Distribution Shift via Minimum Discriminating Information

Tobias Sutter, Andreas Krause, Daniel Kuhn

Abstract

Machine learning models often fail when deployed in environments that differ from their training data. While arbitrary distribution shifts are impossible to overcome without assumptions, practitioners often possess partial structural knowledge about the target environment. This paper introduces MDI-DRO, a framework that integrates the Principle of Minimum Discriminating Information (MDI) with Distributionally Robust Optimization (DRO). By combining MDI to model known structural shifts and DRO to account for finite-sample uncertainty, the proposed method achieves robust generalization. We demonstrate the effectiveness of MDI-DRO across synthetic covariate shifts, real-world medical data, and off-policy evaluation in reinforcement learning.

1. Introduction

The fundamental assumption of empirical risk minimization (ERM)—that training and testing data are independent and identically distributed (i.i.d.)—is frequently violated in real-world applications. Whether due to sample selection bias, non-stationary environments, or covariate shifts, these "distribution shifts" can lead to catastrophic performance degradation.

While addressing arbitrary shifts is theoretically impossible, many practical scenarios provide "hints" about the target distribution. For instance, a researcher might know the population-level mean of certain features even if their training sample is biased. The challenge lies in incorporating this partial knowledge into the training process while remaining robust to the inherent noise of finite datasets. This paper presents MDI-DRO, a mathematically rigorous framework designed to bridge this gap.

2. The MDI-DRO Framework

The MDI-DRO approach decomposes the problem of robust generalization into two distinct components: modeling the shift and managing uncertainty.

2.1 Information Projection (MDI)

To incorporate known structural information about the test distribution, the authors utilize the Principle of Minimum Discriminating Information. Given a set of constraints (e.g., known moments or ranges in the test set), MDI identifies the "most natural" test distribution. This is achieved by finding a distribution that satisfies the constraints while minimizing the Kullback-Leibler (KL) divergence from the training distribution. This "least prejudiced" approach ensures that no unnecessary assumptions are made about the shift beyond what is explicitly provided.

2.2 Distributionally Robust Optimization (DRO)

Because training data is finite, the empirical distribution is only an approximation of the true underlying distribution. MDI-DRO employs DRO to create an uncertainty ball (typically using KL divergence) around the empirical data. The model is then trained to minimize the worst-case risk within this ball.

2.3 Synergy and Optimization

The novelty of MDI-DRO lies in the combination: MDI handles the known structural shift, while DRO addresses the unknown sampling error. To make this computationally feasible, the authors developed a "double smoothing" fast gradient algorithm, allowing for efficient optimization of the complex minimax problems inherent in the framework.

3. Empirical Evaluation

The framework was validated across three diverse scenarios to test its versatility.

  • Synthetic Covariate Shift: In a controlled logistic regression environment with six features, MDI-DRO successfully adapted to intentional skews between training and testing sets.
  • Real-World Heart Disease Data: Using the UCI/Kaggle Heart Disease dataset, the authors simulated sample selection bias by over-representing specific demographics (males over 60). MDI-DRO significantly outperformed standard ERM, which failed to generalize to the broader population.
  • Inventory Control (Off-Policy Evaluation): In a reinforcement learning context, the "shift" occurs between a behavioral policy (data collection) and an evaluation policy (target). MDI-DRO provided more accurate and lower-variance cost estimates compared to Inverse Propensity Scoring (IPS), particularly in data-scarce regimes.

4. Key Insights

The research highlights several critical advantages of the MDI-DRO approach:

  1. Statistical Efficiency: The framework provides a tight upper bound on true risk, offering better guarantees than traditional importance sampling methods.
  2. Reduced Information Requirements: Unlike Importance Weighted ERM (IWERM), which requires the exact density ratio of the shift, MDI-DRO only requires partial structural information, such as feature moments.
  3. Versatility: The mathematical foundation applies equally to static classification and sequential decision-making tasks.

5. Limitations and Future Directions

Despite its strengths, MDI-DRO relies on the accuracy of the provided prior knowledge. If the structural assumptions about the test distribution are incorrect, the information projection may lead the model astray. Additionally, the DRO radius ($r$) remains a sensitive hyperparameter that requires careful tuning, often through cross-validation. While the current work assumes i.i.d. training samples, extending these proofs to highly correlated data in complex RL environments remains a promising area for future research.

6. Conclusion

MDI-DRO offers a robust middle ground between standard ERM and purely pessimistic robust optimization. By leveraging partial knowledge through MDI and protecting against sampling noise via DRO, it provides a principled path toward models that can reliably perform under real-world distribution shifts.

Summary updated

Abstract

Training models that perform well under distribution shifts is a central challenge in machine learning. In this paper, we introduce a modeling framework where, in addition to training data, we have partial structural knowledge of the shifted test distribution. We employ the principle of minimum discriminating information to embed the available prior knowledge, and use distributionally robust optimization to account for uncertainty due to the limited samples. By leveraging large deviation results, we obtain explicit generalization bounds with respect to the unknown shifted distribution. Lastly, we demonstrate the versatility of our framework by demonstrating it on two rather distinct applications: (1) training classifiers on systematically biased data and (2) off-policy evaluation in Markov Decision Processes.
BibTeX
@inproceedings{sutter2021robust,
  author = {Tobias Sutter and Andreas Krause and Daniel Kuhn},
  title = {{Robust Generalization despite Distribution Shift via Minimum Discriminating Information}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=kiWRlrbVzSM},
}
Figure