Improving Multi-Domain Generalization through Domain Re-labeling

Kowshik Thopalli, Sameeksha Katoch, Andreas Spanias, Pavan K. Turaga, Jayaraman J. ThiagarajanarXiv 2021 · arXiv · OpenReview

Abstract

Domain generalization (DG) methods aim to develop models that generalize to settings where the test distribution is different from the training data. In this paper, we focus on the challenging problem of multi-source zero shot DG (MDG), where labeled training data from multiple source domains is available but with no access to data from the target domain. A wide range of solutions have been proposed for this problem, including the state-of-the-art multi-domain ensembling approaches. Despite these advances, the naïve ERM solution of pooling all source data together and training a single classifier is surprisingly effective on standard benchmarks. In this paper, we hypothesize that, it is important to elucidate the link between pre-specified domain labels and MDG performance, in order to explain this behavior. More specifically, we consider two popular classes of MDG algorithms — distributional robust optimization (DRO) and multi-domain ensembles, in order to demonstrate how inferring custom domain groups can lead to consistent improvements over the original domain labels that come with the dataset. To this end, we propose (i) Group-DRO++, which incorporates an explicit clustering step to identify custom domains in an existing DRO technique; and (ii) DReaME, which produces effective multi-domain ensembles through implicit domain re-labeling with a novel meta-optimization algorithm. Using empirical studies on multiple standard benchmarks, we show that our variants consistently outperform ERM by significant margins (1.5% - 9%), and produce state-of-the-art MDG performance.

Introduction

In Multi-Source Zero-Shot Domain Generalization (MDG), models are trained on several source domains (e.g., photos, sketches) to perform well on an unseen target domain (e.g., oil paintings). Traditionally, researchers use pre-defined domain labels to learn domain-invariant features. However, a persistent paradox exists: Empirical Risk Minimization (ERM)—which ignores these labels and pools all data—often outperforms complex domain-aware algorithms.

We hypothesize that the "Domain Label Bottleneck" is the culprit. Human-defined categories like "sketch" or "cartoon" may not reflect the actual statistical distribution shifts relevant to a model's generalization. Images within a single source domain might belong to different statistical groups, while images across domains might share more in common than their labels suggest.

Proposed Methods

To overcome the limitations of fixed labels, we propose automated domain discovery through two novel algorithms.

Group-DRO++

Group-DRO++ enhances Distributional Robust Optimization (DRO) by replacing fixed groups with an iterative clustering mechanism. Instead of relying on provided labels, the algorithm extracts latent features and applies K-means to identify "custom" domains. The model is then trained to minimize the error of the worst-performing custom group, with clusters refined periodically throughout training.

The Group-DRO++ pipeline featuring iterative clustering for custom domain discovery.

DReaME (Domain Re-labeling for Multi-Domain Ensembles)

DReaME is a meta-learning approach for building ensembles. Rather than assigning one model per pre-defined domain, DReaME utilizes a gradient-matching strategy. During meta-validation, validation batches are assigned to the ensemble model whose training gradients most closely align with the validation gradients. This allows the ensemble to implicitly reorganize data into groups that optimize for generalization rather than just fitting label-based silos.

The DReaME meta-learning framework for implicit domain re-labeling and ensemble updates.

Experimental Evaluation

We evaluated our methods on five diverse benchmarks: PACS, VLCS, OfficeHome, TerraIncognita, and Camelyon17-WILDS.

Key Results

  • Performance Gains: Our methods consistently outperformed the strong ERM baseline by 1.5% to 9% across various benchmarks.
  • State-of-the-Art: DReaME achieved new state-of-the-art results on PACS (+1.1%), VLCS (+1.2%), and OfficeHome (+1.36%).
  • Real-World Robustness: On the Camelyon17 medical dataset, DReaME reached 94.6% accuracy, surpassing specialized methods like SagNet and CORAL by approximately 2%.
  • Gradient Matching vs. Loss: Empirical findings showed that matching models to data based on gradient direction is more effective than using the lowest loss, suggesting that learning dynamics are a better indicator of domain membership than current performance.

Conclusion and Future Work

Our work demonstrates that moving beyond fixed domain labels is crucial for advancing MDG. By discovering custom groupings, Group-DRO++ and DReaME unlock significant performance improvements. Future research will focus on automatically determining the optimal number of ensemble models and exploring the theoretical foundations of why specific custom groupings enhance generalization. While these methods introduce some computational overhead due to iterative clustering and meta-learning, the gains in robustness make them compelling for critical applications.

Summary updated

Abstract

Domain generalization (DG) methods aim to develop models that generalize to settings where the test distribution is different from the training data. In this paper, we focus on the challenging problem of multi-source zero shot DG (MDG), where labeled training data from multiple source domains is available but with no access to data from the target domain. A wide range of solutions have been proposed for this problem, including the state-of-the-art multi-domain ensembling approaches. Despite these advances, the na\"ive ERM solution of pooling all source data together and training a single classifier is surprisingly effective on standard benchmarks. In this paper, we hypothesize that, it is important to elucidate the link between pre-specified domain labels and MDG performance, in order to explain this behavior. More specifically, we consider two popular classes of MDG algorithms -- distributional robust optimization (DRO) and multi-domain ensembles, in order to demonstrate how inferring custom domain groups can lead to consistent improvements over the original domain labels that come with the dataset. To this end, we propose (i) Group-DRO++, which incorporates an explicit clustering step to identify custom domains in an existing DRO technique; and (ii) DReaME, which produces effective multi-domain ensembles through implicit domain re-labeling with a novel meta-optimization algorithm. Using empirical studies on multiple standard benchmarks, we show that our variants consistently outperform ERM by significant margins (1.5% - 9%), and produce state-of-the-art MDG performance. Our code can be found at https://github.com/kowshikthopalli/DREAME
BibTeX
@misc{thopalli2021improving,
  author = {Kowshik Thopalli and Sameeksha Katoch and Andreas Spanias and Pavan K. Turaga and Jayaraman J. Thiagarajan},
  title = {{Improving Multi-Domain Generalization through Domain Re-labeling}},
  year = {2021},
  eprint = {2112.09802},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.09802},
}
Figure