Statistical Analysis of Conditional Group Distributionally Robust Optimization with Cross-Entropy Loss

Zijian Guo, Zhenyu Wang, Yifan Hu, Francis BacharXiv 2025 · DOI · arXiv

TL;DR

This paper introduces Conditional Group Distributionally Robust Optimization (CG-DRO), a framework for multi-source domain adaptation that learns classifiers robust to shifts between source and target environments. The authors provide the first rigorous statistical analysis of this model under cross-entropy loss, establishing fast convergence rates and a novel perturbation-based inference method to handle "non-standard" scenarios where traditional statistics fail.

What Problem Does This Paper Address?

In machine learning, we often collect data from multiple sources (e.g., different hospitals or geographic regions) to train a model for a new "target" domain. However, these sources often exhibit distributional heterogeneity—the relationship between features ($X$) and labels ($Y$) changes across domains.

Standard Empirical Risk Minimization (ERM) averages these differences, which can lead to poor performance if the target domain looks like the "worst-case" source. While Group DRO exists to minimize this worst-case risk, it typically ignores information from the target domain.

This paper tackles Unsupervised Domain Adaptation, where we have labeled data from sources but only unlabeled covariates from the target. The challenge is twofold:

  1. Theoretical Gap: Most existing DRO theory focuses on squared loss (regression). Classification with cross-entropy loss lacks closed-form solutions, making it much harder to analyze.
  2. Statistical Instability: In minimax (saddle-point) problems like DRO, the resulting estimators often don't follow a "normal" (Gaussian) distribution. This happens due to boundary effects (when one source domain dominates) or system instability (when sources are too similar), rendering standard confidence intervals invalid.

Methods

The authors propose a three-pronged approach:

  • CG-DRO Framework: Unlike standard Group DRO which looks at the joint distribution $P(X, Y)$, CG-DRO focuses on the conditional distribution $P(Y|X)$. By leveraging target covariates, it optimizes for the worst-case mixture of source conditional distributions specifically tailored to the target's feature space.
  • Mirror Prox with Double Machine Learning (DML): To solve the optimization problem, they use the Mirror Prox algorithm (an efficient way to find saddle points). They integrate DML to estimate "nuisance" components—like the density ratio between source and target covariates—ensuring that errors in these estimates don't slow down the model's overall convergence.
  • Perturbation-Based Inference: To solve the "non-normal" distribution problem, they generate a collection of perturbed weight vectors. Instead of relying on a single (potentially biased) estimate, they construct confidence intervals by taking the union of intervals derived from these perturbations. This ensures valid coverage even in "non-regular" or "unstable" regimes.

Datasets

The paper is primarily theoretical and uses carefully designed synthetic datasets to validate its mathematical proofs. These simulations cover:

  • Linear and Non-linear classification: Using models like Logistic Regression and XGBoost.
  • Varying degrees of shift: Controlled by parameters that move the problem from "regular" (easy to estimate) to "non-regular" (boundary effects) and "unstable" (collinear sources).
  • Multi-class settings: Testing the framework beyond simple binary classification.

Key Insights and Ideas

  • The "Aha" Moment on Non-standard Asymptotics: The authors demonstrate that if two source domains are very similar, the "worst-case" weight assigned to them can fluctuate wildly. Similarly, if one domain is clearly the worst, the weight hits a boundary (0 or 1). In both cases, the standard "bell curve" assumption for the estimator collapses.
  • Theoretical Bridges: To prove fast convergence, the authors created "surrogate" minimax problems. These act as mathematical intermediaries that allow them to transfer properties from simpler quadratic problems to the complex cross-entropy landscape.
  • Conditioning Matters: By focusing on $P(Y|X)$ and using target $X$ data, CG-DRO consistently outperforms standard Group DRO, which can be overly conservative because it tries to be robust to shifts in $X$ that we already have data for.

Key Results

  • Fast Convergence: The authors prove that the CG-DRO estimator converges at a rate of $O(\sqrt{d/n})$, where $d$ is the dimension and $n$ is the sample size. This is the "gold standard" parametric rate.
  • Robustness: In simulations, CG-DRO achieved lower worst-case risk than both ERM and standard Group DRO across various source mixtures.
  • Valid Inference: Their perturbation method maintained a 95% coverage rate for confidence intervals even in scenarios where traditional Bootstrap and Normality-based methods dropped to 80% or lower.

Limitations and Future Work

  • Collinearity Sensitivity: While the perturbation method handles it, extreme similarity between source domains still increases the length of confidence intervals, reflecting inherent uncertainty.
  • Computational Cost: Taking the union of many perturbed intervals is more computationally intensive than a single point estimate, though the authors note it is easily parallelizable.
  • High-Dimensional Scaling: While the theory allows for growing dimensions, the current inference justification is strongest for the fixed-dimension regime. Extending the "uniformly valid" inference to extremely high-dimensional settings (where $d > n$) remains an open challenge.

Summary updated

Abstract

In multi-source learning with discrete labels, distributional heterogeneity across domains poses a central challenge to developing predictive models that transfer reliably to unseen domains. We study multi-source unsupervised domain adaptation, where labeled data are available from multiple source domains and only unlabeled data are observed from the target domain. To address potential distribution shifts, we propose a novel Conditional Group Distributionally Robust Optimization (CG-DRO) framework that learns a classifier by minimizing the worst-case cross-entropy loss over the convex combinations of the conditional outcome distributions from sources domains. We develop an efficient Mirror Prox algorithm for solving the minimax problem and employ a double machine learning procedure to estimate the risk function, ensuring that errors in nuisance estimation contribute only at higher-order rates. We establish fast statistical convergence rates for the empirical CG-DRO estimator by constructing two surrogate minimax optimization problems that serve as theoretical bridges. A distinguishing challenge for CG-DRO is the emergence of nonstandard asymptotics: the empirical CG-DRO estimator may fail to converge to a standard limiting distribution due to boundary effects and system instability. To address this, we introduce a perturbation-based inference procedure that enables uniformly valid inference, including confidence interval construction and hypothesis testing.
BibTeX
@misc{guo2025statistical,
  author = {Zijian Guo and Zhenyu Wang and Yifan Hu and Francis Bach},
  title = {{Statistical Analysis of Conditional Group Distributionally Robust Optimization with Cross-Entropy Loss}},
  year = {2025},
  eprint = {2507.09905},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.09905},
  howpublished = {arXiv},
}
Figure