Abstract
Machine learning models deployed in real-world environments frequently encounter distribution shifts and adversarial attacks. While unsupervised domain adaptation (UDA) addresses the former and adversarial training (AT) addresses the latter, combining them remains a significant challenge due to the lack of labels in the target domain. This paper introduces DART (Divergence Aware adveRsarial Training), a theoretically grounded framework for Robust Unsupervised Domain Adaptation (RUDA). By deriving a new generalization bound based on worst-case domain divergence, DART provides a unified strategy that significantly enhances model robustness on unlabeled target data. Experimental results across 46 source-target pairs demonstrate that DART outperforms existing state-of-the-art methods by up to 29% in robust accuracy while maintaining or improving clean accuracy.
Introduction
The intersection of distribution shift and adversarial vulnerability presents a critical hurdle for reliable machine learning. In standard UDA, models are trained on labeled source data and unlabeled target data to overcome domain gaps. However, these models are often highly susceptible to adversarial perturbations in the target domain. Traditional adversarial training requires ground-truth labels to generate adversarial examples, a luxury not available for the target domain in UDA. Previous RUDA methods relied on heuristics; DART addresses this by providing a principled approach derived from a new generalization bound.
Methods
The DART framework is built upon a novel generalization bound which posits that adversarial loss on the target domain is upper-bounded by three components:
- Source Loss: The model's performance on labeled source data.
- Worst-case Domain Divergence: Unlike standard UDA, which minimizes the distance between clean distributions, DART minimizes the divergence between the source and the adversarially perturbed target distribution.
- Ideal Joint Loss: The performance of an optimal classifier across both domains, estimated in DART via dynamic pseudo-labeling.
DART functions as a meta-algorithm, compatible with various UDA backbones such as DANN, MMD, or CORAL. It utilizes a divergence-aware regularizer to ensure the feature space remains invariant even under worst-case target perturbations. To estimate the ideal joint loss without target labels, DART maintains a pseudo-label predictor that is updated periodically based on validation performance.
Results
The authors evaluated DART using the DomainRobust testbed, comprising 46 source-target pairs across DIGIT, OfficeHome, PACS, and VisDA benchmarks. The experimental campaign was extensive, involving the training of nearly 30,000 models. As shown in Table 1, DART consistently achieves superior robust accuracy compared to baselines.
Key findings include:
- Superior Robustness: DART outperformed all baselines, including recent methods like ARTUDA and SRoUDA, across all four benchmarks.
- Significant Gains: On the DIGIT benchmark, DART achieved a relative improvement in robust accuracy of up to 29.2% on specific domain pairs.
- Consistency: DART ranked first in robust target accuracy for 35 out of 46 source-target pairs under the rigorous AutoAttack evaluation.
- No Accuracy Trade-off: Unlike supervised adversarial training, DART frequently improves clean accuracy on the target domain while simultaneously boosting robustness.
Conclusion
DART provides a mathematically rigorous foundation for robust unsupervised domain adaptation. By focusing on the worst-case domain divergence and leveraging dynamic pseudo-labels, it bridges the gap between domain invariance and adversarial robustness. While the computational cost of adversarial training remains a factor, DART's consistent performance across diverse benchmarks establishes it as a robust framework for real-world deployment. Future research directions include extending this principled approach to other tasks like object detection and exploring its applicability to domain generalization.
Summary updated