TL;DR
This paper introduces Constrained Maximum Cross-Domain Likelihood (CMCL), a novel method for domain generalization that aims to learn models robust to unseen data distributions. CMCL achieves this by jointly aligning posterior and marginal distributions across domains, leading to improved generalization performance on various benchmarks.
What Problem Does This Paper Address?
Deep learning models excel when training and testing data come from the same distribution. However, in real-world applications, this assumption often breaks down due to "distribution shift," where data from different sources (e.g., hospitals, cameras) have different statistical properties. This shift can severely degrade model performance. Domain generalization (DG) aims to train a single model that performs well on unseen domains without any prior knowledge of them, making it crucial for reliable AI in diverse environments. Existing DG methods often focus on aligning feature distributions, but they struggle to achieve true joint distribution alignment and can suffer from issues like increased entropy or reliance on unavailable target domain information.
Methods
CMCL tackles domain generalization by proposing a novel optimization objective that jointly aligns posterior and marginal distributions. The core idea is to minimize the KL-divergence between posterior distributions of different domains, aiming for a domain-invariant classifier. To enhance generalization, this expectation is computed on the ground-truth marginal distribution.
To address two key challenges:
- KL-divergence side-effect: Minimizing KL-divergence can lead to increased entropy (uncertainty) in posterior distributions. CMCL introduces a "maximum in-domain likelihood" term to counteract this, reinforcing the discriminative power of the learned features.
- Unavailability of ground-truth marginal distribution: The paper assumes the ground-truth marginal distribution lies within the convex hull of the source domain marginal distributions. This allows approximating the ground-truth marginal distribution using the source domains.
These components are combined into a "Constrained Maximum Cross-Domain Likelihood" (CMCL) optimization problem. An alternating optimization strategy with three stages (Stage A: empirical risk minimization and marginal alignment; Stage B: posterior estimation with frozen feature extractor; Stage C: feature extractor update via maximum cross-domain likelihood) is employed to solve this problem.
Datasets
The proposed method is evaluated on four standard domain generalization benchmarks:
- Digits-DG: A digit recognition benchmark consisting of four datasets (MNIST, MNIST-M, SVHN, SYN).
- PACS: An object recognition benchmark with four domains (Photo, Art, Cartoon, Sketch), characterized by large style discrepancies.
- Office-Home: Another object recognition benchmark with four domains (Art, Clipart, Product, Real-world), featuring shifts in image style and viewpoint.
- miniDomainNet: A larger object recognition benchmark, a subset of DomainNet, containing four domains (Clipart, Painting, Real, Sketch).
Key Insights and Ideas
- Joint alignment is key: The paper emphasizes that aligning both posterior ($P(Y|Z)$) and marginal ($P(Z)$) distributions is crucial for true joint distribution alignment ($P(Z, Y)$) and robust generalization.
- Ground-truth marginal distribution is a powerful concept: Computing expectations over the ground-truth marginal distribution, even if approximated, significantly enhances generalizability compared to using only source domain data.
- Convex hull assumption is practical: The assumption that the true marginal distribution lies within the convex hull of source domain distributions provides a reasonable basis for aligning marginal distributions in DG.
- Maximum in-domain likelihood as a crucial regularizer: This term effectively counteracts the negative entropy side-effect of KL-divergence minimization, preserving classification performance.
Key Results
CMCL consistently outperforms existing domain generalization methods across all tested datasets.
- On Digits-DG, CMCL achieves the best average accuracy, surpassing the second-best method by a significant 3.90%.
- On PACS, CMCL achieves competitive results, especially with a larger backbone (ResNet-50), and demonstrates strong performance on the Sketch and Cartoon domains.
- On Office-Home, CMCL achieves the best average accuracy with ResNet-18 and surpasses all competitors with ResNet-50, improving performance by 4.12% on the Art domain.
- On miniDomainNet, CMCL achieves the best performance on all domains, with a large margin of 3.22% average accuracy improvement.
- Qualitative results (t-SNE visualizations) show that CMCL learns more aligned and discriminative features compared to a baseline (DeepAll), with source and test domain features clustering better by semantic category.
Limitations and Future Work
- Assumption 1's necessity: The paper notes that the performance on MNIST-M in Digits-DG is worse than other domains, suggesting that this domain might not lie within the convex hull of the other source domains. This highlights that Assumption 1 is necessary and reasonable for distribution alignment-based DG.
- Future work: The authors suggest coupling CMCL with generative methods to diversify training domains, making the real-world marginal distribution more likely to fall within the convex hull of the training domains.
Summary updated