TL;DR
This paper introduces Discriminant Risk Minimization (DRM), a novel theory and algorithm for domain generalization that works without requiring domain labels. By minimizing the discrepancy in prediction distributions across different subsets of the source data, DRM encourages models to learn invariant features, leading to improved performance on unseen target domains.
What Problem Does This Paper Address?
Deep learning models often struggle when faced with data from new, unseen domains, a phenomenon known as domain shift. This is frequently because models learn to rely on "spurious features" – characteristics that happen to be correlated with the class label in the training data but aren't truly indicative of the class itself. For example, a model might learn to associate a specific background with a particular object, leading to poor performance if that object appears in a different background in a new domain.
Domain generalization (DG) aims to build models that are robust to such shifts, learning from multiple source domains to perform well on target domains without ever seeing the target data during training. A major hurdle in existing DG methods is their reliance on domain labels, which precisely identify which source domain a data point comes from. However, in many real-world scenarios, these labels are unavailable or costly to obtain. This paper tackles the challenge of achieving domain generalization in a domain-agnostic manner, meaning without needing to know which source domain each training sample belongs to.
Methods
The core of the proposed solution is the Discriminant Risk Minimization (DRM) theory. The key idea is that if a model relies on spurious features, its prediction distribution will vary significantly when looking at different subsets of the source data. Conversely, a model that has learned invariant features will exhibit stable prediction distributions across these subsets. DRM quantifies this discrepancy using a new risk metric.
To implement this theory, the authors propose an algorithm that combines two main components:
- Bayesian Inference: Instead of outputting a single prediction, the model's output layer is replaced with a Bayesian linear layer. This allows the model to output a probability distribution over predictions, inherently capturing uncertainty. This probabilistic output is crucial for aligning with the theoretical assumptions and for approximating prediction distributions.
- Categorical Discriminant Risk (CDR): This is a novel penalty term designed to measure the discrepancy between prediction distributions of different data subsets. To make this computationally feasible, they introduce a "Discriminant matrix" that is updated incrementally using a sliding window approach. This avoids the need to compute distributions over entire datasets at each step.
The optimization objective then becomes minimizing both the standard task loss (like cross-entropy) and the newly defined CDR penalty.
Datasets
The authors evaluate their method on three widely used real-world benchmarks for domain generalization:
- PACS: Contains four sub-datasets (art, cartoon, photo, sketch).
- VLCS: Contains four sub-datasets (Caltech101, LabelMe, SUN09, VOC2007).
- Office-Home: Contains four sub-datasets (art, clipart, product, real).
They use these datasets in a standard DG setting where one domain is held out as the test set, and the remaining are used for training and validation.
Key Insights and Ideas
- Domain Labels are a Bottleneck: The paper strongly argues that the reliance on domain labels in many existing DG methods limits their applicability in real-world scenarios. Their work directly addresses this by proposing a label-free approach.
- Prediction Distribution Discrepancy as a Signal for Spurious Features: The core insight is that the variability of a model's prediction distribution across different subsets of the same source domain is a strong indicator of its reliance on spurious features. Stable distributions imply invariant feature learning.
- Bayesian Uncertainty for Robustness: Using a Bayesian linear layer to output prediction distributions, rather than point estimates, is shown to improve generalization and stability. This probabilistic output is key to approximating the required distributions for the CDR penalty.
- Incremental Update for Efficiency: The introduction of the sliding Discriminant matrix (CDR) makes the theoretically sound DRM approach computationally tractable for deep learning models.
Key Results
The proposed DRM method achieves state-of-the-art or competitive performance across the evaluated datasets, particularly excelling on average.
- On PACS, DRM achieved an accuracy of 82.6 ± 0.4%.
- On VLCS, DRM achieved an accuracy of 76.12 ± 0.9%.
- On Office-Home, DRM achieved an accuracy of 63.03 ± 0.3%, which was only marginally lower than the best-performing method.
Crucially, DRM consistently outperformed many methods that do rely on domain labels, demonstrating the effectiveness of their label-free approach. The paper also highlights that DRM exhibits enhanced stability and robustness, evidenced by lower standard deviations in its results and a relative insensitivity to hyperparameter tuning. Ablation studies showed that removing the Bayesian linear layer or the CDR penalty individually led to a decrease in accuracy and/or increased instability.
Limitations and Future Work
- Computational Cost: While the sliding update for CDR improves efficiency, the paper notes that generating confusion matrices for approximation (an earlier idea) is time-consuming. The Bayesian inference also adds computational overhead.
- Hyperparameter Sensitivity (Minor): While generally robust, the paper mentions that overly large or small values of hyperparameters $\alpha$ (CDR weight) and $\beta$ (sliding update rate) can lead to reduced accuracy.
- Future Directions: The paper suggests that their approach can be extended to promote fairness by maintaining consistency in model outputs. The theoretical framework also opens avenues for further research into understanding and improving domain generalization.
Summary updated